diff --git a/README.md b/README.md index 8cd98527ea..39be4748ee 100644 --- a/README.md +++ b/README.md @@ -56,8 +56,6 @@ functionality preserved. ### Ego Pipeline - Capture egocentric data without a robot in the loop. -- Reconstruct hand and camera poses from egocentric video with the - [Ego Post Processing pipeline](https://nvidia.github.io/IsaacCapture/main/references/egocentric_hand_reconstruction.html). ## Quick Start diff --git a/docs/source/_static/egocentric-input.gif b/docs/source/_static/egocentric-input.gif deleted file mode 100644 index 8e32fde921..0000000000 --- a/docs/source/_static/egocentric-input.gif +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:6b6518c2c9cc650fcf52409f6970aef95de379ece383a8ddc2a3a298d280f1d0 -size 1464915 diff --git a/docs/source/_static/egocentric-reconstruction.gif b/docs/source/_static/egocentric-reconstruction.gif deleted file mode 100644 index 3122b62982..0000000000 --- a/docs/source/_static/egocentric-reconstruction.gif +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:efca9b174082840be7b0aca9302b77730f4870c19b77a12b60db6ac0b3726663 -size 454670 diff --git a/docs/source/index.rst b/docs/source/index.rst index 6702419607..bde0f4d1b0 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -97,7 +97,6 @@ Table of Contents references/cloudxr references/oob_teleop_control references/haptic_feedback - references/egocentric_hand_reconstruction Indices and tables ------------------ diff --git a/docs/source/references/egocentric_hand_reconstruction.rst b/docs/source/references/egocentric_hand_reconstruction.rst deleted file mode 100644 index 236980bd68..0000000000 --- a/docs/source/references/egocentric_hand_reconstruction.rst +++ /dev/null @@ -1,260 +0,0 @@ -.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -.. SPDX-License-Identifier: Apache-2.0 - -Egocentric Hand Reconstruction -============================== - -Automated pipeline for 4D hand and camera pose reconstruction from egocentric -videos. Integrates ViPE and Dyn-HaMR in containerized environments. - -.. list-table:: - :widths: 50 50 - - * - .. image:: ../_static/egocentric-input.gif - :alt: Source egocentric video - :width: 100% - :class: no-image-zoom - - .. image:: ../_static/egocentric-reconstruction.gif - :alt: Smooth fit grid reconstruction - :width: 100% - :class: no-image-zoom - * - .. centered:: Source egocentric video - - .. centered:: Reconstructed 4D hand and camera poses - - -Video Capture ---------------------------- - -To capture egocentric video with an OAK camera, see the -`OAK camera plugin `_ documentation. - -Setup ------ - -System Requirement -^^^^^^^^^^^^^^^^^^ - -- OS: Ubuntu 24.04 -- GPU: NVIDIA RTX 6000 Ada, L40, H100, GeForce RTX 3090, GeForce RTX 4090 -- System RAM: 100GB (for a reference 30s video, more for longer) -- System VRAM: 12GB (for a reference 30s video, more for longer) -- Free Disk: 100GB - -Prerequisites -^^^^^^^^^^^^^ - -Ensure the following are installed and configured before starting: - -**Docker ≥ 20.10** (BuildKit support required): - -.. code-block:: bash - - docker --version # should print 20.10 or newer - -**NVIDIA Container Toolkit** — required for GPU access inside containers: - -- Install guide: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html - -**Python tooling** — required only for downloading videos from S3/Swift URLs: - -.. code-block:: bash - - pip install boto3 - - -Checkout the code -^^^^^^^^^^^^^^^^^ - -.. code-block:: bash - - git clone https://github.com/NVIDIA/IsaacTeleop.git - cd IsaacTeleop/src/postprocessing/egocentric_hand_reconstruction - -The ``./docker`` and ``./scripts`` directories referenced in this guide are located under this directory. - -Prepare data files -^^^^^^^^^^^^^^^^^^ - -Place required files in the ``outputs/`` directory. - -.. code-block:: text - - ... - ├── docker/ - ├── scripts/ - ├── osmo/ - └── outputs/ - ├── MANO_RIGHT.pkl - └── BMC/ - └── *.npy - -**MANO model** (required): - -- Create an academic account at https://mano.is.tue.mpg.de/ and accept the license. -- The download is a ZIP archive — extract it and place ``MANO_RIGHT.pkl`` in ``outputs/``. - -**BMC data** (required): - -- Follow the README in https://github.com/MengHao666/Hand-BMC-pytorch to - generate (until the step ``python calculate_bmc.py``) -- Place all ``.npy`` files in: ``outputs/BMC/`` - -.. note:: - - The Hand-BMC-pytorch repository is no longer actively maintained, so parts - of its setup may not work out-of-the-box on newer systems. At the time of - writing, the ``environment.yml`` pins PyTorch to a specific build - (``py3.7_cuda10.0.130_cudnn7.6.2_0``) that may no longer be available on - Conda channels or compatible with current hardware. If Conda fails to - resolve the environment, one workaround is to relax the pins in - ``environment.yml``: - - .. code-block:: yaml - - # Before - - pytorch==1.2.0=py3.7_cuda10.0.130_cudnn7.6.2_0 - - torchvision==0.4.0=py37_cu100 - - # After - - pytorch=1.2.0 - - torchvision=0.4.0 - - This fix reflects the state of the upstream repo at the time of writing and - may need to be adjusted as the ecosystem evolves. - -Build Docker images -^^^^^^^^^^^^^^^^^^^ - -.. code-block:: bash - - ./docker/vipe.sh build - ./docker/dynhamr.sh build - -.. note:: - - Building these Docker images pulls third-party source code, libraries, and - pre-trained model weights from external repositories. These components are - subject to their own respective licenses, which may include restrictions on - use, modification, or redistribution. It is the user's responsibility to - review and comply with all applicable third-party licenses before building, - using, or distributing these images. Refer to each Dockerfile for the - specific sources pulled during the build. - -Hand Reconstruction -------------------- - -Run complete reconstruction (ViPE + Dyn-HaMR) with a single command: - -.. code-block:: bash - - # Using a local video file - ./scripts/run_reconstruction.sh path/to/your_video.mp4 - - # Using a remote video file - ./scripts/run_reconstruction.sh s3://path/to/your_video.mp4 - -The script accepts either a **local file path** or a remote **URL** -pointing to a video on cloud storage. Both ``s3://`` URLs (S3-compatible -cloud storage) and ``swift://`` URLs (OpenStack Object Storage) are -supported. When a URL is provided, the video is automatically downloaded -to the ``outputs/`` directory before processing begins. - -To use a remote video, set the following environment variables for -credentials: - -.. list-table:: - :header-rows: 1 - :widths: 28 12 60 - - * - Variable - - Required - - Description - * - ``ACCESS_KEY_ID`` - - Yes - - Your S3 access key ID - * - ``SECRET_ACCESS_KEY`` - - Yes - - Your S3 access key - * - ``BUCKET_REGION`` - - No - - Region (default: ``us-east-1``) - * - ``BUCKET_ENDPOINT_URL`` - - No - - Custom endpoint for S3-compatible storage - -By default, the pipeline reads data files from and writes results to the -``outputs/`` directory. Set ``OUTPUTS_DIR`` to use a different location: - -.. code-block:: bash - - OUTPUTS_DIR=/path/to/outputs ./scripts/run_reconstruction.sh path/to/your_video.mp4 - -The pipeline will: - -1. Copy or download the video to ``outputs/``. -2. Run ViPE to estimate camera poses. -3. Run Dyn-HaMR for hand reconstruction. -4. Save all results to ``outputs/logs/``. - -Batch Reconstruction with OSMO -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -For large-scale batch processing, the pipeline can be submitted as an -`OSMO `_ workflow using ``hand_reconstruction.yaml``. -This runs ViPE and Dyn-HaMR as two chained tasks on a GPU pool. - -**Prerequisites:** - -- A working OSMO cluster deployment (see the `OSMO deployment guide `_) -- OSMO CLI installed and authenticated (``osmo login …``) -- Bucket and image registry credentials stored in OSMO -- Container images built and pushed to your registry (see `Build Docker images`_) -- MANO and BMC assets available at an S3 URL - -See ``osmo/README.md`` for full setup details including credential registration and container image push steps. - -**Submit a workflow:** - -.. code-block:: bash - - osmo workflow submit osmo/hand_reconstruction.yaml \ - --pool POOL_NAME \ - --set-string \ - experiment_id=EXPERIMENT_ID \ - source_url=s3://INPUT_S3_PATH \ - dest_url=s3://OUTPUT_S3_PATH \ - assets_url=s3://ASSETS_S3_PATH \ - vipe_image=CONTAINER_REGISTRY/ego_vipe:TAG \ - dynhamr_image=CONTAINER_REGISTRY/ego_dynhamr:TAG - -**Monitor progress:** - -.. code-block:: bash - - osmo workflow logs WORKFLOW_ID -n 100 - -Estimated Runtime -^^^^^^^^^^^^^^^^^ - -For a reference 30-second video, expect approximately: - -- **ViPE**: ~7 minutes -- **Dyn-HaMR**: ~30 minutes - -Actual runtime may vary depending on system hardware and video length. - -View results -^^^^^^^^^^^^ - -.. code-block:: bash - - # List results - ls outputs/logs/video-custom//*/ - - # View visualization - vlc outputs/logs/video-custom//*/*_grid.mp4 - -Limitations ------------ - -The quality of the reconstructed result is directly related to the capture quality of the egocentric video. diff --git a/src/postprocessing/egocentric_hand_reconstruction/docker/Dockerfile.dynhamr b/src/postprocessing/egocentric_hand_reconstruction/docker/Dockerfile.dynhamr deleted file mode 100644 index 090e43dcde..0000000000 --- a/src/postprocessing/egocentric_hand_reconstruction/docker/Dockerfile.dynhamr +++ /dev/null @@ -1,162 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -# NOTE: Building this image pulls third-party source code and pre-trained -# model weights from external repositories. These components are subject to -# their own respective licenses. Review them before use or redistribution. - -FROM nvidia/cuda:12.1.1-devel-ubuntu22.04 - -ENV DEBIAN_FRONTEND=noninteractive -ENV TZ=US/Pacific -RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone - -# Install system dependencies -RUN apt-get update && apt-get install -y --no-install-recommends \ - git ssh ca-certificates curl wget \ - build-essential cmake pkg-config \ - ffmpeg \ - libgl1 libegl1 libglib2.0-0 libxrender1 libsm6 libxext6 libglfw3 \ - && rm -rf /var/lib/apt/lists/* - -# Create non-root user -ARG USERNAME=appuser -ARG USER_UID=1000 -ARG USER_GID=1000 -RUN groupadd --gid $USER_GID $USERNAME \ - && useradd --no-log-init --uid $USER_UID --gid $USER_GID -m $USERNAME - -USER $USERNAME -WORKDIR /home/appuser - -# Install Miniconda -ENV CONDA_PLUGINS_AUTO_ACCEPT_TOS=true -RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \ - && bash Miniconda3-latest-Linux-x86_64.sh -b -p /home/appuser/miniconda3 \ - && rm -f Miniconda3-latest-Linux-x86_64.sh - -ENV PATH="/home/appuser/miniconda3/bin:${PATH}" - -# Initialize conda -RUN conda init bash - -# Create Conda environment for Dyn-HaMR -RUN conda create -n dynhamr python=3.10 -y - -# Activate env by default in interactive shells -RUN echo "conda activate dynhamr" >> /home/appuser/.bashrc - -# Set the default shell to activate the environment -SHELL ["conda", "run", "-n", "dynhamr", "/bin/bash", "-c"] -ENV PATH="/home/appuser/miniconda3/envs/dynhamr/bin:${PATH}" - -# Install PyTorch (matching Dyn-HaMR requirements) -RUN pip install torch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 --index-url https://download.pytorch.org/whl/cu121 - -# Install core dependencies (pin numpy to 1.23.5 for chumpy compatibility) -RUN pip install \ - "numpy==1.23.5" \ - scipy \ - opencv-python \ - matplotlib \ - scikit-image \ - einops \ - hydra-core \ - omegaconf \ - trimesh \ - smplx \ - ultralytics==8.1.34 \ - pyrender \ - pyglet \ - pillow \ - tqdm \ - jupyter \ - ipdb - -# Install additional dependencies discovered during integration. -# Re-pin numpy==1.23.5 here (same value as the core line) to anchor the -# resolver: plyfile 1.1.4 requires numpy>=2.0 and would otherwise upgrade numpy -# past the version chumpy needs. With numpy pinned, pip selects a compatible -# plyfile automatically. -RUN pip install \ - "numpy==1.23.5" \ - joblib \ - loguru \ - plyfile \ - gdown \ - imageio-ffmpeg \ - yacs \ - pytorch-lightning \ - webdataset \ - av \ - braceexpand \ - dill - -# Install manopth and chumpy for MANO hand model -RUN pip install git+https://github.com/hassony2/manopth.git -RUN pip install git+https://github.com/hassony2/chumpy.git - -# Install dependencies required by core pipeline (run_opt.py and HaMeR) -RUN pip install \ - tensorboard \ - timm \ - huggingface_hub - -# NOTE: Other VIPE-specific dependencies (roma, kornia) are NOT needed -# when using pre-computed VIPE results. Only install if you plan to run VIPE camera estimation. - -# Set CUDA_HOME for building CUDA extensions -ENV CUDA_HOME=/usr/local/cuda - -# Clone Dyn-HaMR at pinned commit -ARG DYNHAMR_REPO=https://github.com/ZhengdiYu/Dyn-HaMR.git -ARG DYNHAMR_COMMIT=9d92bf432c214846f3713dc9308192e6accd06af -RUN git clone --recursive ${DYNHAMR_REPO} /home/appuser/Dyn-HaMR && \ - cd /home/appuser/Dyn-HaMR && \ - git checkout ${DYNHAMR_COMMIT} && \ - git submodule update --init --recursive && \ - find /home/appuser/Dyn-HaMR -name ".git" -exec rm -rf {} + 2>/dev/null; true - -# Install Dyn-HaMR from root (setup.py handles subdirectory structure) -WORKDIR /home/appuser/Dyn-HaMR -RUN pip install -e . - -# Copy setup script -ARG CACHEBUST=1 -COPY --chown=$USERNAME:$USERNAME docker/setup_dynhamr.sh /home/appuser/setup_dynhamr.sh - -# Hand-mesh generation entry points (sit in WORKDIR so `from body_model ...`, -# `from util ...`, `from vis ...` resolve against the installed Dyn-HaMR source). -COPY --chown=$USERNAME:$USERNAME \ - mesh_generation/save_hand_mesh_trajectory.py \ - mesh_generation/generate_mesh.sh \ - /home/appuser/Dyn-HaMR/dyn-hamr/ -RUN chmod +x /home/appuser/Dyn-HaMR/dyn-hamr/generate_mesh.sh - -# --------------------------------------------------------------------------- -# Pre-download model assets that are otherwise fetched at runtime. -# -# 1. hamer_demo_data.tar.gz. download_models() in -# hamer/models/__init__.py is called by run.py with -# --checkpoint /home/appuser/Dyn-HaMR and skips the download when -# _DATA/hamer_ckpts already exists, so pre-extracting here is sufficient. -# -# 2. detector.pt — setup_dynhamr.sh already guards this with -# [ ! -f ], so it will be skipped automatically once pre-baked. -# --------------------------------------------------------------------------- -RUN cd /home/appuser/Dyn-HaMR && \ - wget -nv "https://www.cs.utexas.edu/~pavlakos/hamer/data/hamer_demo_data.tar.gz" && \ - tar --warning=no-unknown-keyword --exclude=".*" -xf hamer_demo_data.tar.gz && \ - rm hamer_demo_data.tar.gz - -RUN mkdir -p /home/appuser/Dyn-HaMR/third-party/hamer/pretrained_models && \ - wget -nv -O /home/appuser/Dyn-HaMR/third-party/hamer/pretrained_models/detector.pt \ - "https://huggingface.co/spaces/rolpotamias/WiLoR/resolve/main/pretrained_models/detector.pt" - -# Jumpy-hand QC scripts (lightweight; deps already satisfied by numpy + matplotlib) -COPY --chown=$USERNAME:$USERNAME quality_control/detect_jumpy_hand_from_world_results.py /home/appuser/detect_jumpy_hand_from_world_results.py -COPY --chown=$USERNAME:$USERNAME quality_control/cal_jumpy_proportions.py /home/appuser/cal_jumpy_proportions.py -COPY --chown=$USERNAME:$USERNAME quality_control/run_jumpy_qc.sh /home/appuser/run_jumpy_qc.sh - -# Set working directory to dyn-hamr subdirectory where run_opt.py is located -WORKDIR /home/appuser/Dyn-HaMR/dyn-hamr diff --git a/src/postprocessing/egocentric_hand_reconstruction/docker/Dockerfile.vipe b/src/postprocessing/egocentric_hand_reconstruction/docker/Dockerfile.vipe deleted file mode 100644 index a3275be39a..0000000000 --- a/src/postprocessing/egocentric_hand_reconstruction/docker/Dockerfile.vipe +++ /dev/null @@ -1,110 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -# NOTE: Building this image pulls third-party source code and pre-trained -# model weights from external repositories. These components are subject to -# their own respective licenses. Review them before use or redistribution. - -FROM nvidia/cuda:12.4.0-devel-ubuntu22.04 - -ENV DEBIAN_FRONTEND=noninteractive -ENV TZ=US/Pacific -RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone - -# Install system dependencies -RUN apt-get update && apt-get install -y --no-install-recommends \ - git ssh ca-certificates curl wget \ - build-essential cmake pkg-config \ - ffmpeg \ - libgl1 libegl1 libglib2.0-0 libxrender1 libsm6 libxext6 libglfw3 \ - && rm -rf /var/lib/apt/lists/* - -# Create non-root user -ARG USERNAME=appuser -ARG USER_UID=1000 -ARG USER_GID=1000 -RUN groupadd --gid $USER_GID $USERNAME \ - && useradd --no-log-init --uid $USER_UID --gid $USER_GID -m $USERNAME - -USER $USERNAME -WORKDIR /home/appuser - -# Install Miniconda -ENV CONDA_PLUGINS_AUTO_ACCEPT_TOS=true -RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \ - && bash Miniconda3-latest-Linux-x86_64.sh -b -p /home/appuser/miniconda3 \ - && rm -f Miniconda3-latest-Linux-x86_64.sh - -ENV PATH="/home/appuser/miniconda3/bin:${PATH}" - -# Initialize conda -RUN conda init bash - -# Clone ViPE at pinned commit -ARG VIPE_REPO=https://github.com/nv-tlabs/vipe.git -ARG VIPE_COMMIT=fdc79359ec6d063c86b9d203ca3d7054a9636823 -RUN git clone --recursive ${VIPE_REPO} /home/appuser/ViPE && \ - cd /home/appuser/ViPE && \ - git checkout ${VIPE_COMMIT} && \ - git submodule update --init --recursive && \ - find /home/appuser/ViPE -name ".git" -exec rm -rf {} + 2>/dev/null; true - -# Create Conda environment from ViPE base.yml -WORKDIR /home/appuser/ViPE -RUN conda env create -f envs/base.yml - -# Activate env by default in interactive shells -RUN echo "conda activate vipe" >> /home/appuser/.bashrc - -# Set the default shell to activate the environment -SHELL ["conda", "run", "-n", "vipe", "/bin/bash", "-c"] -ENV PATH="/home/appuser/miniconda3/envs/vipe/bin:${PATH}" - -# Set CUDA_HOME for building CUDA extensions -ENV CUDA_HOME=/usr/local/cuda - -# Install Python dependencies from requirements.txt -RUN pip install -r envs/requirements.txt - -# Set CUDA architectures for compilation (since no GPU available during build) -ENV TORCH_CUDA_ARCH_LIST="7.5;8.0;8.6;8.9;9.0" - -# Install ViPE package with CUDA extensions -RUN pip install --no-build-isolation -e . - -# --------------------------------------------------------------------------- -# Pre-download model weights that ViPE lazily fetches on first inference. -# Without this, every run downloads from GitHub / Google Drive / -# HuggingFace, adding latency and risking rate limits. -# --------------------------------------------------------------------------- - -# Pin cache roots so models are found regardless of the runtime UID. -ENV TORCH_HOME=/home/appuser/.cache/torch -ENV HF_HOME=/home/appuser/.cache/huggingface - -# 1. Direct HTTP models → torch hub cache -RUN mkdir -p ${TORCH_HOME}/hub/geocalib \ - ${TORCH_HOME}/hub/sam \ - ${TORCH_HOME}/hub/checkpoints \ - && wget -nv -O ${TORCH_HOME}/hub/geocalib/pinhole.tar \ - "https://github.com/cvg/GeoCalib/releases/download/v1.0/geocalib-pinhole.tar" \ - && wget -nv -O ${TORCH_HOME}/hub/sam/sam_vit_b_01ec64.pth \ - "https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth" \ - && wget -nv -O ${TORCH_HOME}/hub/checkpoints/groundingdino_swint_ogc.pth \ - "https://huggingface.co/ShilongLiu/GroundingDINO/resolve/main/groundingdino_swint_ogc.pth" \ - && wget -nv -O ${TORCH_HOME}/hub/checkpoints/video_depth_anything_vits.pth \ - "https://huggingface.co/depth-anything/Video-Depth-Anything-Small/resolve/main/video_depth_anything_vits.pth" - -# 2. Google Drive models via gdown (installed from requirements.txt) -RUN mkdir -p ${TORCH_HOME}/hub/aot \ - ${TORCH_HOME}/hub/droid_slam \ - && gdown "https://drive.google.com/uc?id=1QoChMkTVxdYZ_eBlZhK2acq9KMQZccPJ" \ - -O ${TORCH_HOME}/hub/aot/R50_DeAOTL_PRE_YTB_DAV.pth \ - && gdown "https://drive.google.com/uc?id=1PpqVt1H4maBa_GbPJp4NwxRsd9jk-elh" \ - -O ${TORCH_HOME}/hub/droid_slam/droid.pth - -# 3. HuggingFace Hub models (UniDepth V2, Prior-Depth-Anything) -RUN python -c "from huggingface_hub import snapshot_download, hf_hub_download; snapshot_download('lpiccinelli/unidepth-v2-vitl14'); hf_hub_download('Rain729/Prior-Depth-Anything', 'depth_anything_v2_vitb.pth'); hf_hub_download('Rain729/Prior-Depth-Anything', 'prior_depth_anything_vitb.pth')" - -# Set working directory -WORKDIR /home/appuser/ViPE diff --git a/src/postprocessing/egocentric_hand_reconstruction/docker/dynhamr.sh b/src/postprocessing/egocentric_hand_reconstruction/docker/dynhamr.sh deleted file mode 100755 index 73c4fff4f6..0000000000 --- a/src/postprocessing/egocentric_hand_reconstruction/docker/dynhamr.sh +++ /dev/null @@ -1,123 +0,0 @@ -#!/bin/bash -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -# Configuration -IMAGE_NAME="ego_dynhamr:latest" -CONTAINER_NAME="ego-dynhamr" -PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -DOCKERFILE="${PROJECT_ROOT}/docker/Dockerfile.dynhamr" - -# Colors for output -GREEN='\033[0;32m' -RED='\033[0;31m' -YELLOW='\033[1;33m' -NC='\033[0m' # No Color - -# Helper functions -print_info() { - echo -e "${GREEN}[INFO]${NC} $1" -} - -print_warn() { - echo -e "${YELLOW}[WARN]${NC} $1" -} - -print_error() { - echo -e "${RED}[ERROR]${NC} $1" -} - -build_image() { - local no_cache_flag="" - if [[ "$1" == "--clean" ]]; then - no_cache_flag="--no-cache" - print_info "Clean build requested (--no-cache)" - fi - - print_info "Building Dyn-HaMR image: ${IMAGE_NAME}" - - cd "${PROJECT_ROOT}" - - # Get user ID and group ID for non-root container - USER_ID=$(id -u) - GROUP_ID=$(id -g) - - # Build with buildkit for better performance - DOCKER_BUILDKIT=1 docker build ${no_cache_flag} \ - --build-arg USER_UID="${USER_ID}" \ - --build-arg USER_GID="${GROUP_ID}" \ - -t "${IMAGE_NAME}" \ - -f "${DOCKERFILE}" \ - . - - if [[ $? -eq 0 ]]; then - print_info "Build successful!" - else - print_error "Build failed!" - exit 1 - fi -} - -run_container() { - local outputs_dir="${OUTPUTS_DIR:-${PROJECT_ROOT}/outputs}" - print_info "Running container: ${CONTAINER_NAME}" - - # Remove existing container if it exists - if [[ "$(docker ps -aq -f name=${CONTAINER_NAME})" ]]; then - print_info "Removing existing container..." - docker rm -f ${CONTAINER_NAME} > /dev/null - fi - - # Create necessary directories - mkdir -p "${outputs_dir}" - - # Use interactive TTY only when stdin is a terminal - local tty_flags="" - local name_flags="--name ${CONTAINER_NAME}" - if [[ -t 0 ]]; then - tty_flags="-it" - else - name_flags="" - fi - - # Run with GPU support and volume mounts - docker run ${tty_flags} \ - --rm \ - --gpus all \ - --shm-size 16g \ - ${name_flags} \ - -v "${outputs_dir}:/home/appuser/outputs" \ - -v "${outputs_dir}:/home/appuser/Dyn-HaMR/outputs" \ - -w /home/appuser/Dyn-HaMR/dyn-hamr \ - "${IMAGE_NAME}" \ - "$@" -} - -# Main CLI -case "$1" in - "build") - shift - build_image "$@" - ;; - "run") - shift - run_container "$@" - ;; - *) - if [[ -z "$1" ]]; then - # If no argument provided, check if image exists - if [[ "$(docker images -q ${IMAGE_NAME} 2> /dev/null)" == "" ]]; then - print_info "Image not found. Building first..." - build_image - fi - run_container - else - print_info "Usage: $0 {build|run [command]}" - print_info "Examples:" - print_info " $0 build # Build the Docker image" - print_info " $0 build --clean # Build from scratch (no cache)" - print_info " $0 run # Run an interactive shell" - print_info " $0 run python demo.py # Run a specific command" - fi - ;; -esac diff --git a/src/postprocessing/egocentric_hand_reconstruction/docker/setup_dynhamr.sh b/src/postprocessing/egocentric_hand_reconstruction/docker/setup_dynhamr.sh deleted file mode 100755 index f720987c9e..0000000000 --- a/src/postprocessing/egocentric_hand_reconstruction/docker/setup_dynhamr.sh +++ /dev/null @@ -1,108 +0,0 @@ -#!/bin/bash -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -# -# Setup script for Dyn-HaMR data files -# Run this inside the Docker container after first launch - -set -e - -echo "==========================================" -echo "Dyn-HaMR Data Setup" -echo "==========================================" -echo - -# Create directory structure -echo "Creating directory structure..." -mkdir -p /home/appuser/Dyn-HaMR/_DATA/data/mano -mkdir -p /home/appuser/Dyn-HaMR/_DATA/BMC -mkdir -p /home/appuser/Dyn-HaMR/third-party/hamer/pretrained_models -echo "✓ Directories created" -echo - -# Download detector.pt -echo "Downloading YOLO hand detector..." -if [[ ! -f "/home/appuser/Dyn-HaMR/third-party/hamer/pretrained_models/detector.pt" ]]; then - wget https://huggingface.co/spaces/rolpotamias/WiLoR/resolve/main/pretrained_models/detector.pt \ - -P /home/appuser/Dyn-HaMR/third-party/hamer/pretrained_models/ - echo "✓ Detector downloaded" -else - echo "✓ Detector already exists" -fi -echo - -# Copy data from mounted outputs (if available) -echo "Checking for required data files in mounted volumes..." - -if [[ -f "/home/appuser/outputs/MANO_RIGHT.pkl" ]]; then - echo "Copying MANO model..." - cp /home/appuser/outputs/MANO_RIGHT.pkl /home/appuser/Dyn-HaMR/_DATA/data/mano/ - echo "✓ MANO model copied" -else - echo "⚠ MANO_RIGHT.pkl not found in /home/appuser/outputs/" - echo " Please download from: https://mano.is.tue.mpg.de/" - echo " Then copy to: /outputs/MANO_RIGHT.pkl" -fi - -if [[ -d "/home/appuser/outputs/BMC" ]]; then - echo "Copying BMC data..." - cp /home/appuser/outputs/BMC/* /home/appuser/Dyn-HaMR/_DATA/BMC/ - echo "✓ BMC data copied" -else - echo "⚠ BMC data not found in /home/appuser/outputs/" - echo " Please copy BMC files to: /outputs/BMC/" -fi -echo - -# Create placeholder CONVEX_HULLS if needed -if [[ ! -f "/home/appuser/Dyn-HaMR/_DATA/BMC/CONVEX_HULLS.npy" ]]; then - echo "Creating placeholder CONVEX_HULLS.npy..." - python << 'EOF' -import numpy as np -convex_hulls = [np.array([]) for _ in range(21)] -np.save("/home/appuser/Dyn-HaMR/_DATA/BMC/CONVEX_HULLS.npy", convex_hulls, allow_pickle=True) -EOF - echo "✓ CONVEX_HULLS.npy created" -fi -echo - -# Fix config paths -echo "Fixing config paths..." -sed -i 's|root: .*|root: /home/appuser/Dyn-HaMR/test|' \ - /home/appuser/Dyn-HaMR/dyn-hamr/confs/data/video_vipe.yaml -sed -i 's|vipe_dir: .*|vipe_dir: /home/appuser/outputs|' \ - /home/appuser/Dyn-HaMR/dyn-hamr/confs/data/video_vipe.yaml - -# Verify the changes -echo "Verifying config changes:" -grep "vipe_dir:" /home/appuser/Dyn-HaMR/dyn-hamr/confs/data/video_vipe.yaml -echo "✓ Config paths fixed" -echo - -# Disable HMP if not needed -echo "Disabling HMP (Human Motion Prior)..." -cd /home/appuser/Dyn-HaMR/dyn-hamr -if grep -q "^from HMP.fitting import run_prior" run_opt.py; then - # Comment out all HMP-related imports using pattern matching - sed -i 's|^from HMP.fitting import run_prior|# from HMP.fitting import run_prior|' run_opt.py - sed -i 's|^from human_body_prior.tools.model_loader import load_model|# from human_body_prior.tools.model_loader import load_model|' run_opt.py - sed -i 's|^from human_body_prior.models.vposer_model import VPoser|# from human_body_prior.models.vposer_model import VPoser|' run_opt.py - sed -i '/^# from HMP.fitting import run_prior/a run_prior = None # HMP disabled' run_opt.py - echo "✓ HMP disabled in run_opt.py" -else - echo "✓ HMP already disabled" -fi -echo - -echo "==========================================" -echo "Setup Complete!" -echo "==========================================" -echo -echo "Verify data files:" -echo " ls -lh /home/appuser/Dyn-HaMR/_DATA/data/mano/" -echo " ls -lh /home/appuser/Dyn-HaMR/_DATA/BMC/" -echo " ls -lh /home/appuser/Dyn-HaMR/third-party/hamer/pretrained_models/" -echo -echo "Run demo:" -echo " cd /home/appuser/Dyn-HaMR/dyn-hamr" -echo " python run_opt.py data=video_vipe run_opt=True run_vis=True data.seq=demo3 is_static=False" diff --git a/src/postprocessing/egocentric_hand_reconstruction/docker/vipe.sh b/src/postprocessing/egocentric_hand_reconstruction/docker/vipe.sh deleted file mode 100755 index 255a46e4f8..0000000000 --- a/src/postprocessing/egocentric_hand_reconstruction/docker/vipe.sh +++ /dev/null @@ -1,122 +0,0 @@ -#!/bin/bash -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -# Configuration -IMAGE_NAME="ego_vipe:latest" -CONTAINER_NAME="ego-vipe" -PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -DOCKERFILE="${PROJECT_ROOT}/docker/Dockerfile.vipe" - -# Colors for output -GREEN='\033[0;32m' -RED='\033[0;31m' -YELLOW='\033[1;33m' -NC='\033[0m' # No Color - -# Helper functions -print_info() { - echo -e "${GREEN}[INFO]${NC} $1" -} - -print_warn() { - echo -e "${YELLOW}[WARN]${NC} $1" -} - -print_error() { - echo -e "${RED}[ERROR]${NC} $1" -} - -build_image() { - local no_cache_flag="" - if [[ "$1" == "--clean" ]]; then - no_cache_flag="--no-cache" - print_info "Clean build requested (--no-cache)" - fi - - print_info "Building ViPE image: ${IMAGE_NAME}" - - cd "${PROJECT_ROOT}" - - # Get user ID and group ID for non-root container - USER_ID=$(id -u) - GROUP_ID=$(id -g) - - # Build with buildkit for better performance - DOCKER_BUILDKIT=1 docker build ${no_cache_flag} \ - --build-arg USER_UID="${USER_ID}" \ - --build-arg USER_GID="${GROUP_ID}" \ - -t "${IMAGE_NAME}" \ - -f "${DOCKERFILE}" \ - . - - if [[ $? -eq 0 ]]; then - print_info "Build successful!" - else - print_error "Build failed!" - exit 1 - fi -} - -run_container() { - local outputs_dir="${OUTPUTS_DIR:-${PROJECT_ROOT}/outputs}" - print_info "Running container: ${CONTAINER_NAME}" - - # Remove existing container if it exists - if [[ "$(docker ps -aq -f name=${CONTAINER_NAME})" ]]; then - print_info "Removing existing container..." - docker rm -f ${CONTAINER_NAME} > /dev/null - fi - - # Create necessary directories - mkdir -p "${outputs_dir}" - - # Use interactive TTY only when stdin is a terminal - local tty_flags="" - local name_flags="--name ${CONTAINER_NAME}" - if [[ -t 0 ]]; then - tty_flags="-it" - else - name_flags="" - fi - - # Run with GPU support and volume mount - docker run ${tty_flags} \ - --rm \ - --gpus all \ - --shm-size 16g \ - ${name_flags} \ - -v "${outputs_dir}:/home/appuser/ViPE/vipe_results" \ - -w /home/appuser/ViPE \ - "${IMAGE_NAME}" \ - "$@" -} - -# Main CLI -case "$1" in - "build") - shift - build_image "$@" - ;; - "run") - shift - run_container "$@" - ;; - *) - if [[ -z "$1" ]]; then - # If no argument provided, check if image exists - if [[ "$(docker images -q ${IMAGE_NAME} 2> /dev/null)" == "" ]]; then - print_info "Image not found. Building first..." - build_image - fi - run_container - else - print_info "Usage: $0 {build|run [command]}" - print_info "Examples:" - print_info " $0 build # Build the Docker image" - print_info " $0 build --clean # Build from scratch (no cache)" - print_info " $0 run # Run an interactive shell" - print_info " $0 run python run.py # Run a specific command" - fi - ;; -esac diff --git a/src/postprocessing/egocentric_hand_reconstruction/mesh_generation/README_mesh_generation.md b/src/postprocessing/egocentric_hand_reconstruction/mesh_generation/README_mesh_generation.md deleted file mode 100644 index 8f91c640fa..0000000000 --- a/src/postprocessing/egocentric_hand_reconstruction/mesh_generation/README_mesh_generation.md +++ /dev/null @@ -1,137 +0,0 @@ - - -# Hand Mesh Generation - -Export per-track MANO hand meshes and joint trajectories from Dyn-HaMR reconstruction results. -Produces `.npz` files containing vertices, joints, faces, and visibility masks in the final -visualisation world frame — ready to load and plot without a MANO model or any handedness flip. - -Runs inside the Dyn-HaMR Docker image (`docker/Dockerfile.dynhamr`), which already provides -Dyn-HaMR, MANO, and the full Python environment. There is no host-side Python path. - -## Prerequisites - -### 1. Build the Dyn-HaMR image - -```bash -./docker/dynhamr.sh build -``` - -### 2. MANO model - -Place `MANO_RIGHT.pkl` at `outputs/MANO_RIGHT.pkl` (same file used by the reconstruction -pipeline). Download from https://mano.is.tue.mpg.de/ (free registration). The container's -`setup_dynhamr.sh` copies it into `/home/appuser/Dyn-HaMR/_DATA/data/mano/` on first run. - -### 3. Dyn-HaMR results - -Run the full reconstruction pipeline first so each run has `smooth_fit/*_world_results.npz`: - -``` -outputs/logs/video-custom/// -├── track_info.json # optional; per-track vis_mask -└── smooth_fit/ - └── *_world_results.npz -``` - -Mesh export only reads that npz and optional `track_info.json` beside it; no `.hydra/` required. - -## Running - -```bash -./scripts/run_mesh_generation.sh -``` - -Scans `outputs/` by default (override with `OUTPUTS_DIR=…`), finds every run directory -containing `/*_world_results.npz` (default `PHASE=smooth_fit`), and runs the export -once per run on the latest world checkpoint. - -### Environment variables - -| Variable | Default | Description | -|----------------------|----------------|----------------------------------------------------------| -| `OUTPUTS_DIR` | `/outputs` | Host directory mounted into the container at `/home/appuser/outputs` | -| `PHASE` | `smooth_fit` | Phase subdir to read results from | -| `GPU` | `0` | GPU index inside the container | -| `NO_TEMPORAL_SMOOTH` | `0` | Set to `1` to disable OneEuro smoothing (raw poses) | -| `NO_SMOOTH_TRANS` | `0` | Set to `1` to smooth pose only, not translation | - -Example: - -```bash -GPU=1 NO_TEMPORAL_SMOOTH=1 ./scripts/run_mesh_generation.sh -``` - -### Running a single input manually - -To process one `*_world_results.npz` file directly, drop into the container and call the -Python script with its flags: - -```bash -./docker/dynhamr.sh run bash -c " - bash /home/appuser/setup_dynhamr.sh >/dev/null - python /home/appuser/Dyn-HaMR/dyn-hamr/save_hand_mesh_trajectory.py \ - --npz /home/appuser/outputs/logs/.../smooth_fit/_000300_world_results.npz -" -``` - -Flags: - -| Flag | Default | Description | -|-----------------------|----------------|----------------------------------------------------------| -| `--npz` | *(optional)* | Path to `*_world_results.npz`; if omitted, use `--log-dir` + `--phase` | -| `--log-dir` / `--log_dir` | *(with phase)* | Run directory containing `track_info.json` (parent of `smooth_fit/`) | -| `--phase` | `smooth_fit` | Subdir under `--log-dir` that holds `*_world_results.npz` | -| `--iter` | latest | Checkpoint ID when resolving via `--log-dir` (e.g. `000300`) | -| `--fps` | 30 or from `track_info` | FPS written into the output npz | -| `-o / --out` | auto | Output path; default beside source npz: `_hand_mesh_traj_.npz` | -| `--gpu` | `0` | GPU index | -| `--no_temporal_smooth`| off | Disable OneEuro smoothing | -| `--no_smooth_trans` | off | Smooth pose only, not translation | -| `--mano-model-dir` | `/home/appuser/Dyn-HaMR/_DATA/data/mano` | Directory containing `MANO_RIGHT.pkl` (or set `MANO_MODEL_DIR`) | - -## Output - -For each processed log dir, two files are written alongside the world results: - -``` -/smooth_fit/ -├── _hand_mesh_traj_.npz -└── _hand_mesh_traj__meta.json -``` - -### `.npz` arrays - -| Array | Shape | dtype | Description | -|---------------|--------------------|---------|-----------------------------------------------------| -| `verts` | `(B, T, 778, 3)` | float32 | Mesh vertices in world frame | -| `joints` | `(B, T, 21, 3)` | float32 | Joint positions; index 0 = wrist | -| `is_right` | `(B, T)` | int8 | 1 = right hand, 0 = left hand | -| `faces_left` | `(F, 3)` | int32 | Triangle face indices for left-hand meshes | -| `faces_right` | `(F, 3)` | int32 | Triangle face indices for right-hand meshes | -| `vis_mask` | `(B, T)` | int8 | 1 = visible, 0 = occluded, -1 = out of scene | -| `track_ids` | `(B,)` | int | Track IDs | -| `fps` | scalar | float32 | Video frame rate | - -`B` = number of tracks, `T` = sequence length in frames. - -### Loading example - -```python -import numpy as np - -data = np.load("my_seq_hand_mesh_traj_000300.npz") -verts = data["verts"] # (B, T, 778, 3) -joints = data["joints"] # (B, T, 21, 3) -is_right = data["is_right"] # (B, T) -faces_right = data["faces_right"] -faces_left = data["faces_left"] - -# Pick faces for each track based on handedness -for b in range(verts.shape[0]): - faces = faces_right if is_right[b, 0] else faces_left - # verts[b], joints[b], faces are ready to render/plot -``` diff --git a/src/postprocessing/egocentric_hand_reconstruction/mesh_generation/generate_mesh.sh b/src/postprocessing/egocentric_hand_reconstruction/mesh_generation/generate_mesh.sh deleted file mode 100755 index 05c11b11ed..0000000000 --- a/src/postprocessing/egocentric_hand_reconstruction/mesh_generation/generate_mesh.sh +++ /dev/null @@ -1,96 +0,0 @@ -#!/bin/bash -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -# -# In-container mesh generation over Dyn-HaMR world results. Copied into -# /home/appuser/Dyn-HaMR/dyn-hamr/ by docker/Dockerfile.dynhamr. Invoke via -# scripts/run_mesh_generation.sh on the host, not directly on the host. - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" - -RESULTS_DIR="${1:-/home/appuser/outputs}" -PHASE="${PHASE:-smooth_fit}" -GPU="${GPU:-0}" -MANO_MODEL_DIR="${MANO_MODEL_DIR:-/home/appuser/Dyn-HaMR/_DATA/data/mano}" -PYTHON="${PYTHON:-python}" - -EXTRA_ARGS="" -[[ "${NO_TEMPORAL_SMOOTH:-0}" = "1" ]] && EXTRA_ARGS="${EXTRA_ARGS} --no_temporal_smooth" -[[ "${NO_SMOOTH_TRANS:-0}" = "1" ]] && EXTRA_ARGS="${EXTRA_ARGS} --no_smooth_trans" - -if [[ ! -d "${RESULTS_DIR}" ]]; then - echo "Error: Directory not found: ${RESULTS_DIR}" - echo "Usage: $0 [results_dir]" - exit 1 -fi - -if [[ ! -f "${MANO_MODEL_DIR}/MANO_RIGHT.pkl" ]]; then - echo "Error: MANO_RIGHT.pkl not found in ${MANO_MODEL_DIR}/" - echo "Run /home/appuser/setup_dynhamr.sh inside the container to populate it" - echo "(requires MANO_RIGHT.pkl in the mounted outputs/ directory on the host)." - exit 1 -fi - -RESULTS_DIR="$(cd "${RESULTS_DIR}" && pwd)" -echo "==========================================" -echo "Hand mesh generation" -echo "==========================================" -echo "Results dir: ${RESULTS_DIR}" -echo "Phase: ${PHASE}" -echo "GPU: ${GPU}" -echo "MANO models: ${MANO_MODEL_DIR}" -echo "" - -mapfile -t LOG_DIRS < <( - find "${RESULTS_DIR}" -path "*/${PHASE}/*_world_results.npz" -type f 2>/dev/null \ - | while read -r f; do - ( cd "$(dirname "$f")/.." && pwd ) - done | sort -u -) - -if [[ ${#LOG_DIRS[@]} -eq 0 ]]; then - echo "No *_world_results.npz under */${PHASE}/ found under ${RESULTS_DIR}. Nothing to do." - exit 0 -fi - -echo "Found ${#LOG_DIRS[@]} log dir(s)." -echo "" - -SUCCESS=0 -SKIP=0 -FAIL=0 - -for LOG_DIR in "${LOG_DIRS[@]}"; do - PHASE_DIR="${LOG_DIR}/${PHASE}" - if [[ ! -d "${PHASE_DIR}" ]]; then - echo "[SKIP] ${LOG_DIR} (no ${PHASE}/ subdir)" - SKIP=$((SKIP + 1)) - continue - fi - - echo "------------------------------------------" - echo "Processing: ${LOG_DIR}" - if "${PYTHON}" "${SCRIPT_DIR}/save_hand_mesh_trajectory.py" \ - --log-dir "${LOG_DIR}" \ - --phase "${PHASE}" \ - --gpu "${GPU}" \ - --mano-model-dir "${MANO_MODEL_DIR}" \ - ${EXTRA_ARGS}; then - SUCCESS=$((SUCCESS + 1)) - else - echo "WARNING: save_hand_mesh_trajectory.py failed for ${LOG_DIR}" - FAIL=$((FAIL + 1)) - fi -done - -echo "" -echo "==========================================" -echo "Hand mesh generation done." -echo "==========================================" -echo " Succeeded: ${SUCCESS}" -echo " Skipped: ${SKIP} (no ${PHASE}/ subdir)" -echo " Failed: ${FAIL}" -echo "" -echo "Output .npz files written alongside world_results in each /${PHASE}/." diff --git a/src/postprocessing/egocentric_hand_reconstruction/mesh_generation/save_hand_mesh_trajectory.py b/src/postprocessing/egocentric_hand_reconstruction/mesh_generation/save_hand_mesh_trajectory.py deleted file mode 100755 index 48c53795de..0000000000 --- a/src/postprocessing/egocentric_hand_reconstruction/mesh_generation/save_hand_mesh_trajectory.py +++ /dev/null @@ -1,396 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -""" -Export per-track hand meshes and joint trajectories in **final visualization space**: -MANO forward and left-hand x-reflection match `vis.output.prep_result_vis`. - -Loads only `*_world_results.npz` plus optional `track_info.json` in the run directory -(parent of the phase folder, e.g. smooth_fit/). No `.hydra/config.yaml` required. - -Runs inside the Dyn-HaMR Docker image (`docker/Dockerfile.dynhamr`), which installs -Dyn-HaMR at `/home/appuser/Dyn-HaMR/` and sets WORKDIR to `/home/appuser/Dyn-HaMR/dyn-hamr`, -so `body_model`, `util`, and `vis` resolve to the upstream packages. Launch via -`scripts/run_mesh_generation.sh` on the host, which forwards into the container. -""" - -from __future__ import annotations - -# chumpy 0.70 uses APIs removed in newer Python/NumPy; patch before pickle loads MANO. -import numpy as np - -for _alias in ("bool", "int", "float", "complex", "object", "str", "unicode"): - if not hasattr(np, _alias): - setattr( - np, - _alias, - getattr(__builtins__, _alias, None) or getattr(np, f"{_alias}_", None), - ) -import inspect # noqa: E402 - -if not hasattr(inspect, "getargspec"): - inspect.getargspec = inspect.getfullargspec - -import argparse # noqa: E402 -import glob # noqa: E402 -import json # noqa: E402 -import os # noqa: E402 -import re # noqa: E402 -import sys # noqa: E402 - -import torch # noqa: E402 - -from body_model import MANO, run_mano # noqa: E402 -from util.tensor import detach_all, get_device # noqa: E402 -from vis.tools import smooth_results # noqa: E402 - -DEFAULT_MANO_MODEL_DIR = "/home/appuser/Dyn-HaMR/_DATA/data/mano" - - -def load_vis_mask(log_dir: str, B: int, T: int) -> np.ndarray: - """Per-track vis_mask from track_info.json if present, else all ones.""" - ti_path = os.path.join(log_dir, "track_info.json") - if os.path.isfile(ti_path): - try: - with open(ti_path) as f: - info = json.load(f) - tracks = info.get("tracks", {}) - mask = np.ones((B, T), dtype=np.int8) - for b, key in enumerate(sorted(tracks.keys(), key=int)[:B]): - vm = tracks[key].get("vis_mask", None) - if vm is not None: - arr = np.array(vm, dtype=bool) - length = min(len(arr), T) - mask[b, :length] = arr[:length].astype(np.int8) - return mask - except Exception: - pass - return np.ones((B, T), dtype=np.int8) - - -def try_load_fps(log_dir: str, default: float) -> float: - ti_path = os.path.join(log_dir, "track_info.json") - if not os.path.isfile(ti_path): - return default - try: - with open(ti_path) as f: - info = json.load(f) - meta = info.get("meta", {}) - if "fps" in meta: - return float(meta["fps"]) - except Exception: - pass - return default - - -def get_world_results_paths(phase_dir: str) -> dict[str, str]: - """Map iteration id (e.g. 000300) -> path to *_world_results.npz.""" - res_files = sorted(glob.glob(os.path.join(phase_dir, "*_world_results.npz"))) - path_dict: dict[str, str] = {} - for res_file in res_files: - base = os.path.basename(res_file) - parts = base.split("_") - if len(parts) < 4: - continue - it, name, _ = parts[-3:] - if name != "world": - continue - path_dict[it] = res_file - return path_dict - - -def parse_seq_and_iter(npz_path: str) -> tuple[str, str]: - """Infer seq name and iteration from e.g. ``oakd3_000300_world_results.npz``.""" - stem = os.path.splitext(os.path.basename(npz_path))[0] - if "_world_results" in stem: - prefix = stem.split("_world_results")[0] - if "_" in prefix: - seq, it = prefix.rsplit("_", 1) - if it.isdigit(): - return seq, it - return prefix, "000000" - m = re.search(r"_(\d{4,})\.npz$", npz_path) - it = m.group(1) if m else "000000" - return stem, it - - -def numpy_world_to_state(npz_path: str, device: torch.device) -> dict: - """Load world npz arrays as torch tensors on device (pose_body as B x T x 15 x 3).""" - d = np.load(npz_path, allow_pickle=True) - trans = torch.from_numpy(d["trans"].astype(np.float32)).to(device) - root_orient = torch.from_numpy(d["root_orient"].astype(np.float32)).to(device) - pb = d["pose_body"].astype(np.float32) - if pb.ndim == 4: - pose_body = torch.from_numpy(pb).to(device) - else: - pose_body = torch.from_numpy(pb.reshape(pb.shape[0], pb.shape[1], 15, 3)).to( - device - ) - is_right = torch.from_numpy(d["is_right"].astype(np.float32)).to(device) - if "betas" in d.files: - b = d["betas"].astype(np.float32) - if b.ndim == 3: - b = b[:, 0, :] - betas = torch.from_numpy(b).to(device) - else: - betas = None - return { - "trans": trans, - "root_orient": root_orient, - "pose_body": pose_body, - "is_right": is_right, - "betas": betas, - } - - -def _apply_vis_smoothing(res, temporal_smooth: bool, smooth_trans: bool): - """Mirror vis smoothing; pose_body is B x T x 15 x 3.""" - res = detach_all(res) - with torch.no_grad(): - if temporal_smooth: - if smooth_trans: - res["root_orient"], res["pose_body"], res["betas"], res["trans"] = ( - smooth_results( - res["root_orient"], - res["pose_body"], - res["betas"], - res["is_right"], - res["trans"], - ) - ) - else: - res["root_orient"], res["pose_body"], res["betas"], _ = smooth_results( - res["root_orient"], - res["pose_body"], - res["betas"], - res["is_right"], - ) - return res - - -def export_trajectory_pack( - *, - seq_name: str, - iter_name: str, - npz_path: str, - log_dir: str, - device: torch.device, - temporal_smooth: bool, - smooth_trans: bool, - out_path: str, - fps: float, - mano_model_dir: str, -): - res = numpy_world_to_state(npz_path, device) - B, T = res["trans"].shape[:2] - if res["betas"] is None: - res["betas"] = torch.zeros(B, 10, dtype=torch.float32, device=device) - - # MANOLayer sets create_* flags; do not pass create_hand_pose (duplicate kwarg). - hand_model = MANO( - batch_size=B * T, - pose2rot=True, - model_path=mano_model_dir, - use_pca=False, - num_pca_comps=45, - flat_hand_mean=False, - is_rhand=True, - ).to(device) - - res = _apply_vis_smoothing(res, temporal_smooth, smooth_trans) - pose_flat = res["pose_body"].reshape(B, T, -1) - - with torch.no_grad(): - mano_out = run_mano( - hand_model, - res["trans"], - res["root_orient"], - pose_flat, - res["is_right"], - res["betas"], - ) - - verts = mano_out["vertices"].detach().cpu().numpy().astype(np.float32) - joints = mano_out["joints"].detach().cpu().numpy().astype(np.float32) - is_right = mano_out["is_right"].detach().cpu().numpy().astype(np.int8) - faces_left = mano_out["l_faces"].detach().cpu().numpy().astype(np.int32) - faces_right = mano_out["r_faces"].detach().cpu().numpy().astype(np.int32) - - vis_mask = load_vis_mask(log_dir, B, T) - track_ids = np.arange(B, dtype=np.int32) - - meta = { - "format": "dyn_hamr_hand_mesh_traj_v1", - "description": ( - "verts/joints are in the same world frame as run_vis after prep_result_vis. " - "Use faces_left if is_right[b,t]==0 else faces_right." - ), - "source_npz": npz_path, - "seq_name": seq_name, - "iter": iter_name, - "shapes": { - "verts": list(verts.shape), - "joints": list(joints.shape), - "is_right": list(is_right.shape), - "vis_mask": list(vis_mask.shape), - }, - "fps": float(fps), - "temporal_smooth": temporal_smooth, - "smooth_trans": smooth_trans, - "joint_index_wrist": 0, - } - - out_dir = os.path.dirname(os.path.abspath(out_path)) - if out_dir: - os.makedirs(out_dir, exist_ok=True) - np.savez_compressed( - out_path, - verts=verts, - joints=joints, - is_right=is_right, - faces_left=faces_left, - faces_right=faces_right, - vis_mask=vis_mask, - track_ids=track_ids, - fps=np.float32(fps), - ) - meta_path = out_path.replace(".npz", "_meta.json") - with open(meta_path, "w") as f: - json.dump(meta, f, indent=2) - print(f"Wrote {out_path}") - print(f"Wrote {meta_path}") - - -def main(): - parser = argparse.ArgumentParser( - description="Build hand mesh trajectory npz from *_world_results.npz (+ track_info.json)." - ) - parser.add_argument( - "--npz", - default=None, - help="Path to *_world_results.npz (if omitted, use --log-dir + --phase).", - ) - parser.add_argument( - "--log-dir", - "--log_dir", - dest="log_dir", - default=None, - help="Run directory containing track_info.json (parent of smooth_fit/). " - "Default when using --npz: parent of the phase folder.", - ) - parser.add_argument( - "--phase", - default="smooth_fit", - help="Phase subdir under --log-dir when resolving npz (default: smooth_fit).", - ) - parser.add_argument( - "--iter", - default=None, - help="Checkpoint id (e.g. 000300). Default: latest *_world_results in phase dir.", - ) - parser.add_argument( - "-o", - "--out", - default=None, - help="Output .npz path (default: beside source npz: _hand_mesh_traj_.npz).", - ) - parser.add_argument("--gpu", type=int, default=0) - parser.add_argument( - "--fps", - type=float, - default=None, - help="FPS in output (default: 30 or from track_info).", - ) - parser.add_argument( - "--no_temporal_smooth", - action="store_true", - help="Disable OneEuro smoothing.", - ) - parser.add_argument( - "--no_smooth_trans", - action="store_true", - help="Smooth pose only, not translation.", - ) - parser.add_argument( - "--mano-model-dir", - "--mano_model_dir", - dest="mano_model_dir", - default=os.environ.get("MANO_MODEL_DIR", DEFAULT_MANO_MODEL_DIR), - help=( - "Directory containing MANO_RIGHT.pkl. " - f"Default: $MANO_MODEL_DIR or {DEFAULT_MANO_MODEL_DIR}." - ), - ) - args = parser.parse_args() - temporal_smooth = not args.no_temporal_smooth - smooth_trans = not args.no_smooth_trans - device = get_device(args.gpu) - - npz_path = args.npz - log_dir = args.log_dir - - if npz_path: - npz_path = os.path.abspath(npz_path) - if not os.path.isfile(npz_path): - print(f"Not found: {npz_path}", file=sys.stderr) - sys.exit(1) - if log_dir is None: - phase_dir = os.path.dirname(npz_path) - log_dir = os.path.dirname(phase_dir) - else: - log_dir = os.path.abspath(log_dir) - seq_name, iter_name = parse_seq_and_iter(npz_path) - else: - if not args.log_dir: - print( - "Error: pass --npz or --log-dir (with optional --phase).", - file=sys.stderr, - ) - sys.exit(1) - log_dir = os.path.abspath(args.log_dir) - phase_dir = os.path.join(log_dir, args.phase) - if not os.path.isdir(phase_dir): - print(f"Missing phase dir: {phase_dir}", file=sys.stderr) - sys.exit(1) - path_dict = get_world_results_paths(phase_dir) - if not path_dict: - print(f"No *_world_results.npz in {phase_dir}", file=sys.stderr) - sys.exit(1) - if args.iter is not None: - it = args.iter - if it not in path_dict: - print(f"No iteration {it} in {phase_dir}", file=sys.stderr) - sys.exit(1) - else: - it = sorted(path_dict.keys())[-1] - npz_path = path_dict[it] - seq_name, iter_name = parse_seq_and_iter(npz_path) - if args.iter is not None: - iter_name = args.iter - - default_fps = try_load_fps(log_dir, 30.0) - fps = args.fps if args.fps is not None else default_fps - - out_path = args.out - if out_path is None: - out_path = os.path.join( - os.path.dirname(npz_path), - f"{seq_name}_hand_mesh_traj_{iter_name}.npz", - ) - - export_trajectory_pack( - seq_name=seq_name, - iter_name=iter_name, - npz_path=npz_path, - log_dir=log_dir, - device=device, - temporal_smooth=temporal_smooth, - smooth_trans=smooth_trans, - out_path=os.path.abspath(out_path), - fps=fps, - mano_model_dir=args.mano_model_dir, - ) - - -if __name__ == "__main__": - main() diff --git a/src/postprocessing/egocentric_hand_reconstruction/osmo/README.md b/src/postprocessing/egocentric_hand_reconstruction/osmo/README.md deleted file mode 100644 index 8e026166c2..0000000000 --- a/src/postprocessing/egocentric_hand_reconstruction/osmo/README.md +++ /dev/null @@ -1,186 +0,0 @@ - - -# Hand Reconstruction Workflow - -`hand_reconstruction.yaml` is an OSMO workflow that runs a two-stage hand reconstruction pipeline: - -1. **vipe-infer** — Runs ViPE inference on a source video to produce camera pose and depth estimates. -2. **dynhamr-reconstruct** — Feeds the ViPE output into Dyn-HaMR for full hand mesh reconstruction, then runs a jumpy-hand quality control check on the results. - -## Prerequisites - -- OSMO CLI installed and authenticated (`osmo login …`) -- Bucket and image registry credentials stored in OSMO -- An OSMO pool with GPU resources available - -### Data files - -The reconstruction pipeline requires two sets of external data files, stored in an S3 bucket referenced by the `assets_url` parameter (see [Template Parameters](#template-parameters)): - -- **MANO_RIGHT.pkl** -- **BMC/** - -See [`Isaac Teleop Documentation`](https://nvidia.github.io/IsaacCapture/main/references/egocentric_hand_reconstruction.html) for detailed setup instructions. - -### Container images - -The workflow requires two container images (`vipe_image` and `dynhamr_image`). Build them locally following the instructions in [`Isaac Teleop Documentation`](https://nvidia.github.io/IsaacCapture/main/references/egocentric_hand_reconstruction.html): - -```bash -./docker/vipe.sh build -./docker/dynhamr.sh build -``` - -Then push the images to your container registry so OSMO can pull them at runtime: - -```bash -docker tag ego_vipe:latest CONTAINER_REGISTRY/ego_vipe:TAG -docker tag ego_dynhamr:latest CONTAINER_REGISTRY/ego_dynhamr:TAG - -docker push CONTAINER_REGISTRY/ego_vipe:TAG -docker push CONTAINER_REGISTRY/ego_dynhamr:TAG -``` - -### OSMO - -This workflow runs on [OSMO](https://github.com/NVIDIA/OSMO). You need a working OSMO deployment and the OSMO CLI installed. - -#### Deploy OSMO - -Follow the [official deployment guide](https://nvidia.github.io/OSMO/main/deployment_guide/getting_started/infrastructure_setup.html) to set up your own OSMO deployment. - -For a quick single-machine setup, use the [Brev launchable](https://brev.nvidia.com/launchable/deploy/now?launchableID=env-36a6a7qnkOMOP2vgiBRaw2e3jpW). Provision L40S or L40 machines with the Hyperstack provider. - -##### Using the OSMO CLI locally with a Brev deployment - -To interact with a Brev-hosted OSMO instance from your local machine: - -1. **Install the Brev CLI** per the [official guide](https://docs.nvidia.com/brev/latest/cli/getting-started). - -2. **Forward the OSMO port** from your Brev instance (OSMO listens on port 8000): - ```bash - brev port-forward YOUR_BREV_INSTANCE --port LOCAL_PORT:8000 - ``` - -3. **Add a hosts entry** so the CLI and browser can resolve the OSMO hostname: - ```bash - echo "127.0.0.1 quick-start.osmo" | sudo tee -a /etc/hosts - ``` - -4. **Install the OSMO CLI**: - ```bash - curl -fsSL https://raw.githubusercontent.com/NVIDIA/OSMO/refs/heads/main/install.sh | bash - ``` - -5. **Log in**: - ```bash - osmo login http://quick-start.osmo:LOCAL_PORT --method=dev --username=testuser - ``` - -See your Brev instance page for additional Brev CLI commands. - -#### Store credentials in OSMO - -OSMO needs credentials to access the S3 buckets (source data, output destination, assets) and the container image registry. Register them with `osmo credential set`: - -```bash -# S3 bucket access -osmo credential set BUCKET_CREDENTIAL \ - --type DATA \ - --payload \ - endpoint=BUCKET_URL \ - region=us-east-1 \ - access_key_id=ACCESS_KEY_ID \ - access_key=ACCESS_KEY - -# Container registry access -osmo credential set REGISTRY_CREDENTIAL \ - --type REGISTRY \ - --payload \ - registry=REGISTRY_URL \ - username=USERNAME \ - auth=API_KEY -``` - -See the [OSMO credentials documentation](https://nvidia.github.io/OSMO/main/user_guide/getting_started/credentials.html) for details. - -## Template Parameters - -The workflow uses `{{placeholder}}` template variables that are filled at submission time via `--set-string`: - -| Parameter | Required | Description | -|---|---|---| -| `workflow_name` | No | Unique name for the workflow instance. Defaults to `hand-reconstruction`. | -| `experiment_id` | Yes | Folder name of the experiment to process. Must contain at least one `.mp4` file. Appended to `source_url` to form the input path and used to namespace the output under `dest_url`. | -| `source_url` | Yes | S3 URL prefix for source data. The experiment is fetched from `/`. | -| `dest_url` | Yes | S3 URL where outputs are uploaded. Results land at `//…` | -| `assets_url` | Yes | S3 URL for shared reconstruction assets. Must contain `MANO_RIGHT.pkl` and a `BMC/` directory. | -| `vipe_image` | Yes | Container image for the ViPE inference task. | -| `dynhamr_image` | Yes | Container image for the Dyn-HaMR reconstruction task. | - -## Usage - -### Submit a single experiment - -```bash -osmo workflow submit hand_reconstruction.yaml \ - --pool POOL_NAME \ - --set-string \ - workflow_name=hand-rec-my-experiment \ - experiment_id=EXPERIMENT_ID \ - source_url=s3://INPUT_S3_PATH \ - dest_url=s3://OUTPUT_S3_PATH \ - assets_url=s3://ASSETS_S3_PATH \ - vipe_image=CONTAINER_REGISTRY/ego_vipe:TAG \ - dynhamr_image=CONTAINER_REGISTRY/ego_dynhamr:TAG -``` - -### Monitor a running workflow - -```bash -osmo workflow logs WORKFLOW_ID -n 100 -``` - -This streams the trailing 100 lines and continues to follow new output. - -### Check workflow status - -Browse the OSMO dashboard or use: - -```bash -osmo workflow list --name hand-rec --status RUNNING -``` - -## Pipeline Data Flow - -``` -source_url/experiment_id/ - └── *.mp4 ← input video - └── *.hdf5 ← optional sensor data (not used) - - ┌─────────────┐ - │ vipe-infer │ → ViPE pose estimates - └──────┬──────┘ - │ - ┌──────────┴──────────┐ - │ dynhamr-reconstruct │ + assets_url (MANO_RIGHT.pkl, BMC/) - └──────────┬──────────┘ - │ - ▼ -dest_url/experiment_id/ - ├── logs/ ← Dyn-HaMR reconstruction logs - │ └── video-custom/// - │ ├── smooth_fit/*_world_results.npz - │ ├── cameras.json - │ ├── track_info.json - │ ├── *_smooth_fit*grid*.mp4 - │ ├── *_src_cam.mp4 - │ └── qc_results/ - ├── vipe_results/ ← ViPE output - ├── *.hdf5 ← sensor data (passthrough) - ├── video_file.txt - └── video_name.txt -``` diff --git a/src/postprocessing/egocentric_hand_reconstruction/osmo/hand_reconstruction.yaml b/src/postprocessing/egocentric_hand_reconstruction/osmo/hand_reconstruction.yaml deleted file mode 100644 index d3ec58cd36..0000000000 --- a/src/postprocessing/egocentric_hand_reconstruction/osmo/hand_reconstruction.yaml +++ /dev/null @@ -1,180 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -# -# ── Template parameters (supplied via --set-string) ────────────────── -# -# workflow_name Unique OSMO workflow instance name. -# Default: "hand-reconstruction" (see default-values). -# experiment_id Folder name of the experiment to process. Appended to -# source_url for input; namespaces the output under dest_url. -# Must contain at least one .mp4 video file. -# source_url S3 URL prefix for source data. Each experiment is fetched -# from /. -# dest_url S3 URL where final outputs are uploaded. Results land at -# //… -# assets_url S3 URL for shared reconstruction assets. Must contain -# MANO_RIGHT.pkl and a BMC/ directory. -# vipe_image Container image for the ViPE inference task. -# dynhamr_image Container image for the Dyn-HaMR reconstruction task. -# ───────────────────────────────────────────────────────────────────── - -workflow: - name: "{{workflow_name}}" - - resources: - default: - cpu: 15 - gpu: 1 - memory: 100Gi - storage: 200Gi - - tasks: - - name: vipe-infer - image: "{{vipe_image}}" - command: ["bash", "-c"] - args: - - | - set -euo pipefail - - INPUT_DIR="{{input:0}}" - EXPERIMENT_ID="{{experiment_id}}" - VIDEO_PATH="$(find "${INPUT_DIR}" -name '*.mp4' -type f | head -1)" - if [ -z "${VIDEO_PATH}" ]; then - echo "ERROR: No .mp4 file found under ${INPUT_DIR}" - find "${INPUT_DIR}" -type f - exit 1 - fi - VIDEO_FILE="$(basename "${VIDEO_PATH}")" - VIDEO_NAME="${VIDEO_FILE%.*}" - - echo "Experiment: ${EXPERIMENT_ID}" - echo "Using video: ${VIDEO_FILE}" - echo "Video stem: ${VIDEO_NAME}" - - mkdir -p /home/appuser/ViPE/vipe_results - cp "${VIDEO_PATH}" "/home/appuser/ViPE/vipe_results/${VIDEO_FILE}" - - cd /home/appuser/ViPE - vipe infer "vipe_results/${VIDEO_FILE}" - - mkdir -p "{{output}}" - cp "${VIDEO_PATH}" "{{output}}/${VIDEO_FILE}" - cp -r /home/appuser/ViPE/vipe_results/* "{{output}}/" - echo "${VIDEO_FILE}" > "{{output}}/video_file.txt" - echo "${VIDEO_NAME}" > "{{output}}/video_name.txt" - cp "${INPUT_DIR}/"*.hdf5 "{{output}}/" 2>/dev/null || true - - echo "ViPE task complete" - inputs: - - url: "{{source_url}}/{{experiment_id}}" - - - name: dynhamr-reconstruct - image: "{{dynhamr_image}}" - command: ["bash", "-c"] - args: - - | - set -euo pipefail - - PIPELINE_INPUT="{{input:0}}" - ASSETS_DIR="{{input:1}}" - - VIDEO_FILE="$(cat "${PIPELINE_INPUT}/video_file.txt")" - VIDEO_NAME="$(cat "${PIPELINE_INPUT}/video_name.txt")" - EXPERIMENT_ID="{{experiment_id}}" - - echo "Reconstructing: ${VIDEO_FILE} (experiment: ${EXPERIMENT_ID})" - - mkdir -p /home/appuser/outputs - cp -r "${PIPELINE_INPUT}/"* /home/appuser/outputs/ - - # Required assets for Dyn-HaMR setup script. - if [ -f "${ASSETS_DIR}/MANO_RIGHT.pkl" ]; then - cp "${ASSETS_DIR}/MANO_RIGHT.pkl" /home/appuser/outputs/MANO_RIGHT.pkl - else - echo "ERROR: Missing ${ASSETS_DIR}/MANO_RIGHT.pkl" - exit 1 - fi - - if [ -d "${ASSETS_DIR}/BMC" ]; then - mkdir -p /home/appuser/outputs/BMC - cp -r "${ASSETS_DIR}/BMC/"* /home/appuser/outputs/BMC/ - else - echo "ERROR: Missing ${ASSETS_DIR}/BMC directory" - exit 1 - fi - - # Replicate the local dual volume-mount that maps $PROJECT_ROOT/outputs - # to both /home/appuser/outputs AND /home/appuser/Dyn-HaMR/outputs. - # Dyn-HaMR writes logs to ../outputs/logs (relative to dyn-hamr/), - # i.e. /home/appuser/Dyn-HaMR/outputs/logs. - rm -rf /home/appuser/Dyn-HaMR/outputs - ln -sf /home/appuser/outputs /home/appuser/Dyn-HaMR/outputs - - bash /home/appuser/setup_dynhamr.sh - - ln -sf "/home/appuser/outputs/${VIDEO_FILE}" "/home/appuser/Dyn-HaMR/test/videos/${VIDEO_FILE}" - echo "Linked video into Dyn-HaMR test/videos" - - cd /home/appuser/Dyn-HaMR/dyn-hamr - python run_opt.py \ - data=video_vipe \ - run_opt=True \ - run_vis=True \ - data.seq="${VIDEO_NAME}" \ - is_static=False - - # Write results under / so that OSMO uploads them to - # //… preserving the per-experiment layout. - OUTPUT_DIR="{{output}}/${EXPERIMENT_ID}" - mkdir -p "${OUTPUT_DIR}" - if [ -d /home/appuser/outputs/logs ]; then - LOGS_SRC="/home/appuser/outputs/logs" - LOGS_DEST="${OUTPUT_DIR}/logs" - COPIED=0 - - # Copy only essential files from each run directory. - # Structure: logs/video-custom/// - for RUN_DIR in "${LOGS_SRC}"/video-custom/*/*/; do - [ -d "${RUN_DIR}" ] || continue - REL_DIR="${RUN_DIR#${LOGS_SRC}/}" - DEST_RUN="${LOGS_DEST}/${REL_DIR}" - mkdir -p "${DEST_RUN}/smooth_fit" - - cp "${RUN_DIR}/smooth_fit/"*_000300_world_results.npz "${DEST_RUN}/smooth_fit/" 2>/dev/null && COPIED=$((COPIED+1)) - cp "${RUN_DIR}/cameras.json" "${DEST_RUN}/" 2>/dev/null && COPIED=$((COPIED+1)) - cp "${RUN_DIR}/track_info.json" "${DEST_RUN}/" 2>/dev/null && COPIED=$((COPIED+1)) - cp ${RUN_DIR}/*_smooth_fit*grid*.mp4 "${DEST_RUN}/" 2>/dev/null && COPIED=$((COPIED+1)) - cp ${RUN_DIR}/*_src_cam.mp4 "${DEST_RUN}/" 2>/dev/null && COPIED=$((COPIED+1)) - cp -r "${RUN_DIR}/qc_results" "${DEST_RUN}/" 2>/dev/null && COPIED=$((COPIED+1)) - done - - COPIED=$(find "${LOGS_DEST}" -type f 2>/dev/null | wc -l) - if [ "${COPIED}" -gt 0 ]; then - echo "Selectively copied ${COPIED} essential output files to ${LOGS_DEST}" - else - echo "WARN: no essential files found; copying all logs as fallback" - cp -r "${LOGS_SRC}" "${OUTPUT_DIR}/" - fi - fi - if [ -d /home/appuser/outputs/vipe_results ]; then - cp -r /home/appuser/outputs/vipe_results "${OUTPUT_DIR}/" - fi - # Save the original sensor data. - cp /home/appuser/outputs/*.hdf5 "${OUTPUT_DIR}/" 2>/dev/null || true - echo "${VIDEO_FILE}" > "${OUTPUT_DIR}/video_file.txt" - echo "${VIDEO_NAME}" > "${OUTPUT_DIR}/video_name.txt" - - # Jumpy-hand QC on reconstruction results - echo "Running jumpy-hand QC on ${OUTPUT_DIR} ..." - cd /home/appuser - bash run_jumpy_qc.sh "${OUTPUT_DIR}" || echo "WARN: jumpy QC failed (non-fatal)" - - echo "Dyn-HaMR task complete for experiment: ${EXPERIMENT_ID}" - inputs: - - task: vipe-infer - - url: "{{assets_url}}" - outputs: - - url: "{{dest_url}}" - -default-values: - workflow_name: hand-reconstruction diff --git a/src/postprocessing/egocentric_hand_reconstruction/quality_control/README_cal_jumpy_proportions.md b/src/postprocessing/egocentric_hand_reconstruction/quality_control/README_cal_jumpy_proportions.md deleted file mode 100644 index 2a81d08fc4..0000000000 --- a/src/postprocessing/egocentric_hand_reconstruction/quality_control/README_cal_jumpy_proportions.md +++ /dev/null @@ -1,112 +0,0 @@ - - -# cal_jumpy_proportions.py - -Compute **true/false proportions** and **useful frames/seconds** from a jumpy-hand batch summary produced by `detect_jumpy_hand_from_world_results.py`. - ---- - -## What it does - -- **Input:** A `qc_world_jumpy_batch_summary.json` file (from running the detector with `--world_results_dir`). -- **Output:** - - **Proportions:** For each mask key (e.g. `left_hand_translation_mask`, `filtered_valid_mask`), the fraction of frames that are `true` vs `false`, aggregated over all videos. - - **Useful frames/seconds:** Count of frames where `filtered_valid_mask` is `true`, and (if `--fps` is set) the equivalent duration in seconds. - -Statistics are reported in two ways: - -1. **With error videos** — all videos in the summary. -2. **Excluding error videos** — only videos where `error_video` is `false`. - ---- - -## Requirements - -Run the jumpy-hand detector in batch mode first so that `qc_world_jumpy_batch_summary.json` exists: - -```bash -python quality_control/detect_jumpy_hand_from_world_results.py \ - --world_results_dir /path/to/your/results \ - --json --plot --trans_factor 20.0 --rot_factor 20.0 \ - --error_threshold_factor_trans 10.0 --error_threshold_factor_rot 10.0 --error_valid_ratio 0.3 -``` - -Then run this script on that summary (or on the same directory). - ---- - -## Usage - -**Point to the summary file directly:** - -```bash -python quality_control/cal_jumpy_proportions.py --summary /path/to/qc_world_jumpy_batch_summary.json -``` - -**Point to the folder that contains the summary** (script looks for `qc_world_jumpy_batch_summary.json` inside it): - -```bash -python quality_control/cal_jumpy_proportions.py --world_results_dir /path/to/your/results -``` - -**Save the report as JSON:** - -```bash -python quality_control/cal_jumpy_proportions.py --summary /path/to/qc_world_jumpy_batch_summary.json --out proportions.json -``` - -**Include useful seconds** (requires frame rate): - -```bash -python quality_control/cal_jumpy_proportions.py --summary /path/to/qc_world_jumpy_batch_summary.json --fps 30 --out proportions.json -``` - ---- - -## Options - -| Option | Description | -|--------|-------------| -| `--summary PATH` | Path to `qc_world_jumpy_batch_summary.json`. | -| `--world_results_dir DIR` | Directory containing `qc_world_jumpy_batch_summary.json` (summary path = `DIR/qc_world_jumpy_batch_summary.json`). | -| `--out PATH` | Write the full report (proportions + useful stats) to this JSON file. | -| `--fps FLOAT` | Frames per second. If set, report `total_useful_seconds` and `total_seconds` in addition to frame counts. | - -You must provide either `--summary` or `--world_results_dir`. - ---- - -## Output - -### Printed (stdout) - -- Number of videos (total, excluding errors, error count). -- **Useful (filtered_valid_mask = true):** useful frames and total frames for “with error videos” and “excluding error videos”; if `--fps` is set, useful seconds and total seconds. -- Two tables of **proportions** (with / excluding error videos): for each mask key, `proportion_true`, `proportion_false`, and total frames. - -### JSON (with `--out`) - -The report object includes: - -- `summary_path`, `num_videos_all`, `num_videos_excluding_error`, `num_error_videos` -- `useful_frames_with_error_videos`, `useful_frames_excluding_error_videos` — each with `total_frames`, `total_useful_frames`, and (if `--fps` was set) `total_seconds`, `total_useful_seconds` -- `fps` — value passed to `--fps` or `null` -- `proportions_with_error_videos`, `proportions_excluding_error_videos` — per-key stats: `total_frames`, `total_true`, `total_false`, `proportion_true`, `proportion_false` - ---- - -## Mask keys - -The mask keys come from the batch summary’s `frames_false_by_key` (same as in `detect_jumpy_hand_from_world_results.py`), e.g.: - -- `left_hand_translation_mask`, `right_hand_translation_mask` -- `left_hand_rotation_mask`, `right_hand_rotation_mask` -- `left_hand_*_jump_interpolate_mask`, `right_hand_*_jump_interpolate_mask` -- `left_hand_detection_mask`, `right_hand_detection_mask` -- `left_hand_left_of_right_hand_mask` -- `valid_mask`, `filtered_valid_mask` - -See [README_jumpy_hand_filtering.md](README_jumpy_hand_filtering.md) for what each mask means. diff --git a/src/postprocessing/egocentric_hand_reconstruction/quality_control/README_jumpy_hand_filtering.md b/src/postprocessing/egocentric_hand_reconstruction/quality_control/README_jumpy_hand_filtering.md deleted file mode 100644 index 60817756b6..0000000000 --- a/src/postprocessing/egocentric_hand_reconstruction/quality_control/README_jumpy_hand_filtering.md +++ /dev/null @@ -1,220 +0,0 @@ - - -# Jumpy Hand Filtering & Per-Frame Labels - -This document describes the filtering and labeling process implemented in `detect_jumpy_hand_from_world_results.py`. The script analyzes Dyn-HaMR `*_world_results.npz` outputs to detect unstable (“jumpy”) hand tracks and produces per-frame boolean masks you can use for quality control or downstream filtering. - ---- - -## What We Detect - -We label frames along three kinds of issues: - -### 1. Translation jump (per hand) - -- **Definition:** A large change in hand **position** between consecutive frames. -- **Signal:** Adjacent-frame delta of `trans` (world-space root translation), as the L2 norm `‖trans[t] − trans[t−1]‖`. -- **Jump:** The delta exceeds a threshold (see [Thresholds](#thresholds)). - -### 2. Rotation jump (per hand) - -- **Definition:** A large change in hand **orientation** between consecutive frames. -- **Signal:** The relative rotation angle between `root_orient[t−1]` and `root_orient[t]` (axis-angle representation), i.e. the angle of `R[t−1]ᵀ R[t]`. -- **Jump:** That angle (in radians) exceeds a threshold. - -### 3. Detection lost (per hand) - -- **Definition:** The hand was not visible or not detected in that frame. -- **Source:** Not from the world_results NPZ; it comes from **`track_info.json`** in the same run directory. Each track has a `vis_mask`: `true` = visible/detected, `false` = lost. We expose this as a “detection” mask (see below). - -Left and right hands are treated separately (using `is_right` from the NPZ, or track index when that is missing). - ---- - -## Mask Convention - -All per-frame outputs use the same convention: - -- **`true`** = frame is **OK** (no jump, hand detected, or after filtering considered valid). -- **`false`** = frame is **bad** (jump, detection lost, or invalid after filtering). - -So you can use the masks as “valid” flags: keep or use frames where the mask is `true`, and discard or downweight where it is `false`. - ---- - -## Frame index: output frames ≠ raw video frames - -**Frame 0 in the world_results NPZ and in the per-frame labels is not raw video frame 0.** The NPZ and labels use the **sequence interval** defined in `track_info.json`: `meta.seq_interval` is `[start, end)` in raw video frame indices. Output frame index `t` corresponds to **raw video frame `start + t`**. For example, if `seq_interval` is `[22, 949]`, then output frame 0 = raw frame 22, and the last output frame (index 926) = raw frame 948. When mapping labels or masks back to the original video, use `raw_frame = seq_interval[0] + t`. - ---- - -## Pipeline: From Raw Detection to Final Valid Mask - -The script builds several layers of masks. Conceptually: - -``` -Raw jump detection (per hand) - → *_mask (frame-level: true = no jump, false = jump) - → *_jump_interpolate_mask (fill 10-frame gaps between two jump frames with false) -Detection lost from track_info - → *_detection_mask (true = detected, false = lost) - ↓ -valid_mask = AND of all 10 base masks (trans/rot + interpolated + detection, both hands) - ↓ -filtered_valid_mask = valid_mask with “single false” spikes removed (see below) -``` - -So: - -1. **Raw jump masks** - For each hand we get `left/right_hand_translation_mask` and `left/right_hand_rotation_mask`: `true` when that frame does **not** exceed the jump threshold. - -2. **Jump-interpolate masks** - For each of those four jump masks we build a `*_jump_interpolate_mask`: if frame `t` is a jump and there is another jump within the **next 10 frames** at `t'`, then every frame from `t` to `t'` is set to `false`. This fills short “normal” gaps between two nearby jumps so that the whole segment is labeled as invalid. - -3. **Detection masks** - From `track_info.json`, we get `left_hand_detection_mask` and `right_hand_detection_mask`: `true` when the hand is detected (visible) in that frame, `false` when it is lost. - -4. **Valid mask** - `valid_mask[t] = true` only when **all** of the following are `true` at frame `t`: - - `left_hand_translation_mask`, `left_hand_rotation_mask` - - `right_hand_translation_mask`, `right_hand_rotation_mask` - - `left_hand_translation_jump_interpolate_mask`, `left_hand_rotation_jump_interpolate_mask` - - `right_hand_translation_jump_interpolate_mask`, `right_hand_rotation_jump_interpolate_mask` - - `left_hand_detection_mask`, `right_hand_detection_mask` - So a frame is “valid” only if both hands are detected and pass both raw and interpolated jump checks. - -5. **Filtered valid mask** - `filtered_valid_mask` is derived from `valid_mask` by treating **single-frame false spikes** as `true`: if at frame `t` we have `valid_mask[t] = false` but **both** the previous 10 and the next 10 frames are all `true`, then we set `filtered_valid_mask[t] = true`. This removes isolated one-frame invalid labels that are often noise. - ---- - -## Per-Frame Label Keys (in `qc_world_jumpy_per_frame_labels.json`) - -| Key | Meaning | -|-----|--------| -| `left_hand_translation_mask` | `true` = no translation jump at this frame (left hand). | -| `left_hand_rotation_mask` | `true` = no rotation jump at this frame (left hand). | -| `right_hand_translation_mask` | Same for right hand translation. | -| `right_hand_rotation_mask` | Same for right hand rotation. | -| `left_hand_translation_jump_interpolate_mask` | Like above, but 10-frame gaps between two jump frames are also set to `false`. | -| `left_hand_rotation_jump_interpolate_mask` | Same for left rotation. | -| `right_hand_translation_jump_interpolate_mask` | Same for right translation. | -| `right_hand_rotation_jump_interpolate_mask` | Same for right rotation. | -| `left_hand_detection_mask` | `true` = left hand detected (from `track_info.json`). | -| `right_hand_detection_mask` | `true` = right hand detected (from `track_info.json`). | -| `valid_mask` | `true` only when all of the above 10 masks are `true` at this frame. | -| `filtered_valid_mask` | Like `valid_mask`, but single-frame `false` spikes (surrounded by 10 `true` on each side) are set to `true`. | - -Each key is an array of booleans, one per frame index `0..T-1`. - ---- - -## Output Files - -### Per-run (single file or each video in batch) - -- **`qc_world_jumpy_per_frame_labels.json`** - Contains all per-frame mask arrays above. Written next to the world_results (or under `--out_dir` / `--out_subdir`). - -- **`qc_world_jumpy_hand.json`** (optional, with `--json`) - Summary report: selected “jumpy” track, thresholds, jump counts and frame lists, etc. - -- **`qc_world_jumpy_deltas.png`** (optional, with `--plot`) - Plots of per-frame translation and rotation deltas with thresholds. - -### Batch only (when using `--world_results_dir`) - -- **`qc_world_jumpy_batch_summary.json`** - Written at the root of the folder you passed to `--world_results_dir`. It includes: - - **Per video:** `world_results`, `video_id`, `num_frames`, `frames_false_by_key`, `trans_thr`, `rot_thr_deg`, and `error_video` (see below). - - **Folder-level:** `median_trans_threshold`, `median_rot_threshold_deg`, `error_threshold_factor_trans`, `error_threshold_factor_rot`, and `error_video_ids`. - - **Error videos:** A video is labeled `error_video: true` (and its `video_id` added to `error_video_ids`) if any of: - - `trans_thr > median_trans × error_threshold_factor_trans` (default 10.0), or - - `rot_thr_deg > median_rot_deg × error_threshold_factor_rot` (default 10.0), or - - More than 30% of frames are invalid: `filtered_invalid_ratio > error_valid_ratio` (default 0.3). Here `filtered_invalid_ratio` = (number of frames where `filtered_valid_mask` is false) / total_frames. - - Use `--error_threshold_factor_trans`, `--error_threshold_factor_rot`, and `--error_valid_ratio` to change these limits. Per-video `filtered_invalid_ratio` is stored in the summary. - -### Merging per-result summaries (`merge_jumpy_batch_summaries.py`) - -If you run jumpy detection **per result** (e.g. one `qc_world_jumpy_batch_summary.json` per run folder), you can merge them into one overall summary: - -- **`merge_jumpy_batch_summaries.py`** - Finds all `qc_world_jumpy_batch_summary.json` under a base directory, concatenates their `videos` lists, recomputes **median** trans/rot thresholds and **error_video** over the full set, and writes a single `qc_world_jumpy_batch_summary.json` at the base (or at `--out`). - - By default, each video’s **`video_id`** is set to the **folder name that contains** that summary (e.g. `2025-09-04-15-09-41-gr00t005-00013`), so error videos are identifiable by run folder. - ---- - -## Thresholds - -- **Translation:** By default the threshold is `median(translation deltas) × trans_factor`. Default `--trans_factor` is 20.0. You can override with `--trans_abs_threshold` (same units as `trans`). -- **Rotation:** Similarly, default is `median(rotation deltas) × rot_factor` (default 20.0). Override with `--rot_abs_threshold_deg` (degrees). - -So “jump” = frame-to-frame delta above the chosen threshold. Tuning `--trans_factor` and `--rot_factor` (e.g. 20.0 for stricter labeling) controls sensitivity. - ---- - -## Detection Lost and `track_info.json` - -`left_hand_detection_mask` and `right_hand_detection_mask` are built from **`track_info.json`** in the run directory (typically `.../xxx-all-shot-0-0--1/track_info.json`). Each track has a `vis_mask` (one boolean per **raw** frame). The script uses **`meta.seq_interval`** when present: output frame `t` is taken from `vis_mask[seq_start + t]`, so detection masks align with the same sequence range as the NPZ (see [Frame index: output frames ≠ raw video frames](#frame-index-output-frames--raw-video-frames)). If `seq_interval` is missing, the first `num_frames` entries of `vis_mask` are used. We map tracks to left/right via `is_right` in the world_results NPZ (or by track index if `is_right` is missing). If `track_info.json` is not found, both detection masks are set to all `false` (all frames considered “lost” for that hand). You can point to a different file with `--track_info`. - ---- - -## How to Run - -**Single world_results file:** - -```bash -python quality_control/detect_jumpy_hand_from_world_results.py \ - --world_results /path/to/smooth_fit/ego_view_000300_world_results.npz \ - --json --plot --out_dir /path/to/jumpy_hand \ - --trans_factor 20.0 --rot_factor 20.0 -``` - -**Batch (all matching videos under a folder):** - -Only files matching -`logs/video-custom/.../...-all-shot-0-0--1/smooth_fit/*_000300_world_results.npz` -under the given directory are processed. Outputs go under each run’s folder into a subdir named by `--out_subdir` (default `qc_results`). A single summary file is written at the root of the folder. - -```bash -python quality_control/detect_jumpy_hand_from_world_results.py \ - --world_results_dir /path/to/first_50 \ - --json --plot --trans_factor 20.0 --rot_factor 20.0 --error_threshold_factor_trans 10.0 --error_threshold_factor_rot 1.1 --error_valid_ratio 0.3 -``` - -Summary will be at `/path/to/first_50/qc_world_jumpy_batch_summary.json`, and each video’s per-frame labels at -`/path/to/first_50//logs/video-custom/.../...-all-shot-0-0--1/qc_results/qc_world_jumpy_per_frame_labels.json`. - -**Per-result then merge (when you run detection separately per result):** - -If you run detection per result so that each result has its own `qc_world_jumpy_batch_summary.json`, merge them into one overall summary with: - -```bash -python quality_control/merge_jumpy_batch_summaries.py --world_results_dir /path/to/qc_jumpy_samples -``` - -- Discovers all `qc_world_jumpy_batch_summary.json` under the given directory (recursive). -- By default each video’s `video_id` is the **folder name** containing that summary (e.g. `2025-09-04-15-09-41-gr00t005-00013`). -- Writes `qc_world_jumpy_batch_summary.json` at the base (or `--out `). You can then run `quality_control/cal_jumpy_proportions.py --world_results_dir /path/to/qc_jumpy_samples` (or `--summary `) on the merged file. - -Optional: `--error_threshold_factor_trans`, `--error_threshold_factor_rot`, `--error_valid_ratio` override the values used when recomputing `error_video`; otherwise the first summary’s params are used. - -**Building a batch summary from existing `qc_results` only (no NPZ):** -If you have per-result `qc_results` dirs (with `qc_world_jumpy_per_frame_labels.json`) but no per-result batch summary files, you can still produce one summary without re-running on NPZ: -`python quality_control/detect_jumpy_hand_from_world_results.py --build_batch_from_qc_results --qc_results_base_dir /path/to/outputs` -See the script’s `--help` and docstring for details. - ---- - -## Quick Reference: “Which mask should I use?” - -- **Strict:** Use `valid_mask` — keep only frames where every condition (both hands, no translation/rotation jump including interpolated, and detected) is satisfied. -- **Relaxed:** Use `filtered_valid_mask` — same as above but single-frame false spikes are treated as valid. -- **Per-hand / per-type:** Use the individual `*_mask` or `*_jump_interpolate_mask` and `*_detection_mask` keys to filter or analyze by hand or by kind of failure (translation vs rotation vs detection lost). diff --git a/src/postprocessing/egocentric_hand_reconstruction/quality_control/cal_jumpy_proportions.py b/src/postprocessing/egocentric_hand_reconstruction/quality_control/cal_jumpy_proportions.py deleted file mode 100755 index 056ea6e696..0000000000 --- a/src/postprocessing/egocentric_hand_reconstruction/quality_control/cal_jumpy_proportions.py +++ /dev/null @@ -1,239 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -""" -Compute true/false proportions for jumpy-hand mask keys from a batch summary. - -Reads qc_world_jumpy_batch_summary.json and reports, for each mask key: - - With all videos: proportion of frames that are true vs false. - - Excluding error videos: same proportions over only videos with error_video=false. - -Useful frames/seconds: "useful" = frames where filtered_valid_mask is true. - total_useful_frames = sum over videos of (num_frames - # frames where filtered_valid_mask is false). - total_useful_seconds = total_useful_frames / fps (only when --fps is provided). - -Usage: - python quality_control/cal_jumpy_proportions.py --summary /path/to/qc_world_jumpy_batch_summary.json - python quality_control/cal_jumpy_proportions.py --world_results_dir /path/to/folder [--fps 30] - python quality_control/cal_jumpy_proportions.py --summary ... --out proportions.json --fps 30 -""" - -from __future__ import annotations - -import argparse -import json -from pathlib import Path -from typing import Dict, List, Any - - -def load_summary(path: Path) -> Dict[str, Any]: - with open(path, "r") as f: - return json.load(f) - - -def compute_proportions( - videos: List[Dict], - keys: List[str], -) -> Dict[str, Dict[str, float]]: - """ - For each key, compute total_frames, total_true, total_false and proportions. - Returns dict key -> {total_frames, total_true, total_false, proportion_true, proportion_false}. - """ - out: Dict[str, Dict[str, float]] = {} - for key in keys: - total_frames = 0 - total_false = 0 - for v in videos: - n = v.get("num_frames", 0) - false_list = v.get("frames_false_by_key", {}).get(key, []) - total_frames += n - total_false += len(false_list) - total_true = total_frames - total_false - out[key] = { - "total_frames": float(total_frames), - "total_true": float(total_true), - "total_false": float(total_false), - "proportion_true": total_true / total_frames if total_frames else 0.0, - "proportion_false": total_false / total_frames if total_frames else 0.0, - } - return out - - -def compute_useful_frames_and_seconds( - videos: List[Dict], - valid_key: str = "filtered_valid_mask", - fps: float | None = None, -) -> Dict[str, float]: - """ - Useful = frames where valid_key is true. - Returns dict with total_frames, total_useful_frames, and optionally total_seconds, total_useful_seconds. - """ - total_frames = 0 - total_useful_frames = 0 - for v in videos: - n = v.get("num_frames", 0) - false_list = v.get("frames_false_by_key", {}).get(valid_key, []) - total_frames += n - total_useful_frames += n - len(false_list) - out: Dict[str, float] = { - "total_frames": float(total_frames), - "total_useful_frames": float(total_useful_frames), - } - if fps is not None and fps > 0: - out["total_seconds"] = total_frames / fps - out["total_useful_seconds"] = total_useful_frames / fps - return out - - -def main() -> None: - ap = argparse.ArgumentParser( - description="Compute true/false proportions from jumpy batch summary." - ) - ap.add_argument( - "--summary", - type=str, - default=None, - help="Path to qc_world_jumpy_batch_summary.json.", - ) - ap.add_argument( - "--world_results_dir", - type=str, - default=None, - help="Folder containing qc_world_jumpy_batch_summary.json (summary path = /qc_world_jumpy_batch_summary.json).", - ) - ap.add_argument( - "--out", - type=str, - default=None, - help="Optional path to write proportions JSON. If not set, only prints to stdout.", - ) - ap.add_argument( - "--fps", - type=float, - default=None, - help="Frames per second. If set, report total_useful_seconds and total_seconds (useful = filtered_valid_mask true).", - ) - args = ap.parse_args() - - if args.summary is not None: - summary_path = Path(args.summary).expanduser().resolve() - elif args.world_results_dir is not None: - summary_path = ( - Path(args.world_results_dir).expanduser().resolve() - / "qc_world_jumpy_batch_summary.json" - ) - else: - ap.error("Provide either --summary or --world_results_dir.") - - if not summary_path.is_file(): - raise FileNotFoundError(f"Summary file not found: {summary_path}") - - data = load_summary(summary_path) - videos: List[Dict] = data.get("videos", []) - if not videos: - print("No videos in summary.") - return - - # Infer mask keys from first video's frames_false_by_key - keys = sorted(videos[0].get("frames_false_by_key", {}).keys()) - if not keys: - print("No mask keys in summary.") - return - - all_videos = videos - exclude_error = [v for v in videos if not v.get("error_video", False)] - - proportions_all = compute_proportions(all_videos, keys) - proportions_exclude_error = ( - compute_proportions(exclude_error, keys) - if exclude_error - else { - k: { - "total_frames": 0.0, - "total_true": 0.0, - "total_false": 0.0, - "proportion_true": 0.0, - "proportion_false": 0.0, - } - for k in keys - } - ) - - # Useful = frames where filtered_valid_mask is true - useful_all = compute_useful_frames_and_seconds( - all_videos, valid_key="filtered_valid_mask", fps=args.fps - ) - useful_exclude_error = compute_useful_frames_and_seconds( - exclude_error, valid_key="filtered_valid_mask", fps=args.fps - ) - - report = { - "summary_path": str(summary_path), - "num_videos_all": len(all_videos), - "num_videos_excluding_error": len(exclude_error), - "num_error_videos": len(all_videos) - len(exclude_error), - "useful_frames_with_error_videos": useful_all, - "useful_frames_excluding_error_videos": useful_exclude_error, - "fps": args.fps, - "proportions_with_error_videos": proportions_all, - "proportions_excluding_error_videos": proportions_exclude_error, - } - - # Print summary table - print(f"Summary: {summary_path}") - print( - f"Videos: {len(all_videos)} total, {len(exclude_error)} excluding error, {len(all_videos) - len(exclude_error)} error" - ) - print() - print("Useful (filtered_valid_mask = true):") - print( - f" With error videos: {int(useful_all['total_useful_frames']):>10} useful frames / {int(useful_all['total_frames']):>10} total frames", - end="", - ) - if args.fps is not None and args.fps > 0: - print( - f" -> {useful_all['total_useful_seconds']:.2f} s useful / {useful_all['total_seconds']:.2f} s total (fps={args.fps})" - ) - else: - print() - print( - f" Excl. error videos: {int(useful_exclude_error['total_useful_frames']):>10} useful frames / {int(useful_exclude_error['total_frames']):>10} total frames", - end="", - ) - if args.fps is not None and args.fps > 0: - print( - f" -> {useful_exclude_error['total_useful_seconds']:.2f} s useful / {useful_exclude_error['total_seconds']:.2f} s total (fps={args.fps})" - ) - else: - print() - if args.fps is None: - print(" (Pass --fps to get useful seconds and total seconds.)") - print() - print("Proportions (with error videos):") - print(f" {'Mask key':<50} {'prop_true':>12} {'prop_false':>12} total_frames") - print(" " + "-" * 78) - for key in keys: - p = proportions_all[key] - print( - f" {key:<50} {p['proportion_true']:>12.4f} {p['proportion_false']:>12.4f} {int(p['total_frames']):>10}" - ) - print() - print("Proportions (excluding error videos):") - print(f" {'Mask key':<50} {'prop_true':>12} {'prop_false':>12} total_frames") - print(" " + "-" * 78) - for key in keys: - p = proportions_exclude_error[key] - print( - f" {key:<50} {p['proportion_true']:>12.4f} {p['proportion_false']:>12.4f} {int(p['total_frames']):>10}" - ) - - if args.out: - out_path = Path(args.out).expanduser().resolve() - out_path.parent.mkdir(parents=True, exist_ok=True) - with open(out_path, "w") as f: - json.dump(report, f, indent=2) - print(f"\nSaved: {out_path}") - - -if __name__ == "__main__": - main() diff --git a/src/postprocessing/egocentric_hand_reconstruction/quality_control/detect_jumpy_hand_from_world_results.py b/src/postprocessing/egocentric_hand_reconstruction/quality_control/detect_jumpy_hand_from_world_results.py deleted file mode 100755 index 83b4b51f25..0000000000 --- a/src/postprocessing/egocentric_hand_reconstruction/quality_control/detect_jumpy_hand_from_world_results.py +++ /dev/null @@ -1,899 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -""" -Detect the "jumpy" hand/track from a Dyn-HaMR `*_world_results.npz`. - -We consider two independent jump signals (per track): - - Translation jump: large adjacent-frame delta of `trans` (L2 norm). - - Rotation jump: large adjacent-frame delta of `root_orient` (axis-angle), measured as - the relative rotation angle between consecutive frames. - -Outputs: - - Prints a short report to stdout (selected track + stats). - - Optionally writes JSON and a PNG plot. - - Always writes qc_world_jumpy_per_frame_labels.json with per-frame mask arrays - (*_mask: true = normal/detected, false = jump/lost). Includes *_jump_interpolate_mask - for each jump mask: fills 10-frame gaps between two jump frames with false. - Keys: left/right_hand_translation_mask, *_rotation_mask, *_detection_mask, - *_translation_jump_interpolate_mask, *_rotation_jump_interpolate_mask, - and valid_mask (true only when all other masks are true for that frame). - Detection masks come from track_info.json (see --track_info). - -Example (single file): - python quality_control/detect_jumpy_hand_from_world_results.py \\ - --world_results outputs/logs/.../smooth_fit/ego_view_3_000300_world_results.npz \\ - --json --plot --out_dir .../qc_results --trans_factor 3.0 --rot_factor 3.0 - -Example (batch: only logs/video-custom/.../...-all-shot-0-0--1/smooth_fit/*_000300_world_results.npz): - python quality_control/detect_jumpy_hand_from_world_results.py \\ - --world_results_dir /path/to/osmo_results \\ - --json --plot --trans_factor 20.0 --rot_factor 20.0 - (Outputs go to .parent.parent / qc_results; use --out_subdir to change.) - Also writes qc_world_jumpy_batch_summary.json in the folder: per-video listing of which frames have which mask keys false.) - -Default thresholds are median * factor (per track): --trans_factor (default 20.0), - --rot_factor (default 20.0). Or set absolute: --trans_abs_threshold, --rot_abs_threshold_deg. - Batch error flags: --error_threshold_factor_trans, --error_threshold_factor_rot (default 10.0 each), --error_valid_ratio (default 0.3). -""" - -from __future__ import annotations - -import argparse -import json -import math -from dataclasses import dataclass -from pathlib import Path -from typing import Dict, List, Optional, Tuple - -import numpy as np - - -def _nanstat(x: np.ndarray, stat: str) -> float: - if stat == "median": - return float(np.nanmedian(x)) - if stat == "mean": - return float(np.nanmean(x)) - raise ValueError(f"Unknown stat '{stat}'. Expected 'median' or 'mean'.") - - -def _skew(k: np.ndarray) -> np.ndarray: - """ - k: (N,3) - returns: (N,3,3) - """ - kx, ky, kz = k[:, 0], k[:, 1], k[:, 2] - z = np.zeros_like(kx) - K = np.stack( - [ - np.stack([z, -kz, ky], axis=-1), - np.stack([kz, z, -kx], axis=-1), - np.stack([-ky, kx, z], axis=-1), - ], - axis=-2, - ) - return K - - -def axis_angle_to_rotmat(aa: np.ndarray, eps: float = 1e-8) -> np.ndarray: - """ - Convert axis-angle vectors to rotation matrices using Rodrigues formula. - - aa: (N,3) axis-angle in radians. - returns: (N,3,3) - """ - aa = np.asarray(aa, dtype=np.float64) - if aa.ndim != 2 or aa.shape[1] != 3: - raise ValueError(f"axis_angle_to_rotmat expects (N,3), got {aa.shape}") - - theta = np.linalg.norm(aa, axis=1) # (N,) - k = np.zeros_like(aa) - mask = theta > eps - k[mask] = aa[mask] / theta[mask, None] - - K = _skew(k) # (N,3,3) - K2 = np.einsum("nij,njk->nik", K, K) - - sin_t = np.sin(theta)[:, None, None] - cos_t = np.cos(theta)[:, None, None] - - eye = np.eye(3, dtype=np.float64)[None, :, :] - R = eye + sin_t * K + (1.0 - cos_t) * K2 - return R.astype(np.float32) - - -def adjacent_translation_delta(trans_bt3: np.ndarray) -> np.ndarray: - """ - trans_bt3: (B,T,3) - returns: d_bt (B,T) where d[:,0]=NaN and d[:,t]=||trans[t]-trans[t-1]|| for t>=1 - """ - trans = np.asarray(trans_bt3, dtype=np.float32) - if trans.ndim != 3 or trans.shape[-1] != 3: - raise ValueError(f"Expected trans (B,T,3), got {trans.shape}") - d = np.full(trans.shape[:2], np.nan, dtype=np.float32) - dt = trans[:, 1:, :] - trans[:, :-1, :] - d[:, 1:] = np.linalg.norm(dt, axis=-1) - return d - - -def adjacent_rotation_angle_delta(root_orient_bt3: np.ndarray) -> np.ndarray: - """ - root_orient_bt3: (B,T,3) axis-angle (radians) - returns: ang_bt (B,T) where ang[:,0]=NaN and ang[:,t]=angle(R_{t-1}^T R_t) for t>=1 (radians) - """ - aa = np.asarray(root_orient_bt3, dtype=np.float32) - if aa.ndim != 3 or aa.shape[-1] != 3: - raise ValueError(f"Expected root_orient (B,T,3), got {aa.shape}") - B, T, _ = aa.shape - ang = np.full((B, T), np.nan, dtype=np.float32) - - R = axis_angle_to_rotmat(aa.reshape(-1, 3)).reshape(B, T, 3, 3) # (B,T,3,3) - R_prev = R[:, :-1, :, :] - R_cur = R[:, 1:, :, :] - R_prev_T = np.transpose(R_prev, (0, 1, 3, 2)) - R_delta = np.einsum("btij,btjk->btik", R_prev_T, R_cur) # (B,T-1,3,3) - tr = R_delta[..., 0, 0] + R_delta[..., 1, 1] + R_delta[..., 2, 2] - cosang = (tr - 1.0) * 0.5 - cosang = np.clip(cosang, -1.0, 1.0) - ang[:, 1:] = np.arccos(cosang).astype(np.float32) - return ang - - -def _is_right_from_is_right_bt( - is_right_bt: Optional[np.ndarray], b: int -) -> Optional[bool]: - if is_right_bt is None: - return None - x = float(np.asarray(is_right_bt)[b, 0]) - # Most logs store 0/1. Some codebases store track id; for B=2, this still works. - return bool(x > 0.5) - - -@dataclass -class TrackJumpStats: - track_index: int - is_right: Optional[bool] - trans_base: Optional[float] - trans_thr: Optional[float] - trans_jump_count: int - trans_jump_frames: List[int] - trans_max: Optional[float] - rot_base: Optional[float] - rot_thr: Optional[float] - rot_jump_count: int - rot_jump_frames: List[int] - rot_max_rad: Optional[float] - rot_max_deg: Optional[float] - score: float - - -def _frames_where_over(d: np.ndarray, thr: float) -> List[int]: - if not np.isfinite(thr): - return [] - idx = np.where(np.isfinite(d) & (d > float(thr)))[0] - return [int(i) for i in idx.tolist()] - - -def detect_jumpy_tracks( - trans_bt3: np.ndarray, - root_orient_bt3: np.ndarray, - is_right_bt: Optional[np.ndarray], - *, - trans_stat: str, - trans_factor: float, - trans_abs_threshold: Optional[float], - rot_stat: str, - rot_factor: float, - rot_abs_threshold_deg: Optional[float], -) -> Tuple[List[TrackJumpStats], int]: - dtrans_bt = adjacent_translation_delta(trans_bt3) # (B,T) - drot_bt = adjacent_rotation_angle_delta(root_orient_bt3) # (B,T) radians - B = dtrans_bt.shape[0] - - out: List[TrackJumpStats] = [] - for b in range(B): - dtrans = dtrans_bt[b] - drot = drot_bt[b] - - # Translation threshold - trans_base = None - if trans_abs_threshold is None: - if np.any(np.isfinite(dtrans)): - trans_base = _nanstat(dtrans, trans_stat) - trans_thr = ( - None if trans_base is None else float(trans_base) * float(trans_factor) - ) - else: - trans_thr = float(trans_abs_threshold) - - # Rotation threshold (convert degrees -> radians for internal compare) - rot_base = None - if rot_abs_threshold_deg is None: - if np.any(np.isfinite(drot)): - rot_base = _nanstat(drot, rot_stat) - rot_thr_rad = ( - None if rot_base is None else float(rot_base) * float(rot_factor) - ) - else: - rot_thr_rad = math.radians(float(rot_abs_threshold_deg)) - - trans_frames = ( - _frames_where_over(dtrans, trans_thr) if trans_thr is not None else [] - ) - rot_frames = ( - _frames_where_over(drot, rot_thr_rad) if rot_thr_rad is not None else [] - ) - - trans_max = float(np.nanmax(dtrans)) if np.any(np.isfinite(dtrans)) else None - rot_max_rad = float(np.nanmax(drot)) if np.any(np.isfinite(drot)) else None - rot_max_deg = None if rot_max_rad is None else float(np.degrees(rot_max_rad)) - - # Simple combined score: counts + a small tie-breaker on max magnitude. - score = float(len(trans_frames) + len(rot_frames)) - if trans_max is not None: - score += 1e-3 * float(trans_max) - if rot_max_deg is not None: - score += 1e-3 * float(rot_max_deg) - - out.append( - TrackJumpStats( - track_index=b, - is_right=_is_right_from_is_right_bt(is_right_bt, b), - trans_base=trans_base if trans_abs_threshold is None else None, - trans_thr=trans_thr, - trans_jump_count=len(trans_frames), - trans_jump_frames=trans_frames, - trans_max=trans_max, - rot_base=rot_base if rot_abs_threshold_deg is None else None, - rot_thr=None if rot_thr_rad is None else float(rot_thr_rad), - rot_jump_count=len(rot_frames), - rot_jump_frames=rot_frames, - rot_max_rad=rot_max_rad, - rot_max_deg=rot_max_deg, - score=score, - ) - ) - - # Choose best track by score (deterministic tie-breakers embedded in score). - best = int(np.argmax([t.score for t in out])) if len(out) else -1 - return out, best - - -def _maybe_plot( - out_dir: Path, - dtrans_bt: np.ndarray, - drot_bt: np.ndarray, - stats: List[TrackJumpStats], -) -> None: - import matplotlib - - matplotlib.use("Agg") - import matplotlib.pyplot as plt - - B, T = dtrans_bt.shape - x = np.arange(T, dtype=np.int32) - - fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(20, 10), sharex=True) - for b in range(B): - label = f"track {b}" - if stats[b].is_right is not None: - label += " (right)" if stats[b].is_right else " (left)" - (line1,) = ax1.plot(x, dtrans_bt[b], linewidth=1.2, alpha=0.85, label=label) - color = line1.get_color() - ax2.plot( - x, - np.degrees(drot_bt[b]), - linewidth=1.2, - alpha=0.85, - label=label, - color=color, - ) - - if stats[b].trans_thr is not None: - ax1.axhline( - float(stats[b].trans_thr), - linestyle="--", - linewidth=1.0, - alpha=0.35, - color=color, - ) - if stats[b].rot_thr is not None: - ax2.axhline( - float(np.degrees(stats[b].rot_thr)), - linestyle="--", - linewidth=1.0, - alpha=0.35, - color=color, - ) - - # Mark the frames that were actually counted as "jumps" - t_idx = np.asarray(stats[b].trans_jump_frames, dtype=np.int32) - if t_idx.size > 0: - ax1.scatter( - t_idx, - dtrans_bt[b, t_idx], - s=14, - color=color, - alpha=0.75, - edgecolors="none", - marker="o", - ) - r_idx = np.asarray(stats[b].rot_jump_frames, dtype=np.int32) - if r_idx.size > 0: - ax2.scatter( - r_idx, - np.degrees(drot_bt[b, r_idx]), - s=14, - color=color, - alpha=0.75, - edgecolors="none", - marker="o", - ) - - ax1.set_title("Adjacent-frame translation delta ||trans[t]-trans[t-1]||") - ax1.set_ylabel("Distance (world units)") - ax1.grid(True, alpha=0.25) - ax1.legend(loc="upper right", ncol=2) - - ax2.set_title( - "Adjacent-frame rotation delta angle(root_orient[t-1]^T * root_orient[t])" - ) - ax2.set_xlabel("Frame index t (delta is from t-1 → t)") - ax2.set_ylabel("Angle (degrees)") - ax2.grid(True, alpha=0.25) - - fig.tight_layout() - out_path = out_dir / "qc_world_jumpy_deltas.png" - fig.savefig(out_path, dpi=150, bbox_inches="tight") - plt.close(fig) - print(f"Saved plot: {out_path}") - - -def _load_track_info_detection_lost( - track_info_path: Path, num_frames: int, stats: List[TrackJumpStats] -) -> Tuple[List[bool], List[bool]]: - """ - Load track_info.json and return per-frame left_hand_detection_lost, right_hand_detection_lost. - track_info tracks are keyed by track_id; track_id 0 = left, track_id 1 = right (per dataset convention). - Uses meta.seq_interval [start, end) when present: NPZ frame t maps to vis_mask[seq_start + t], so the - sequence length is seq_end - seq_start (should match num_frames). If seq_interval is missing, uses - the first num_frames of vis_mask (legacy behavior). - Returns (left_lost, right_lost), each of length num_frames. If a hand has no track, all True (lost). - """ - left_lost = [True] * num_frames - right_lost = [True] * num_frames - if not track_info_path.is_file(): - return left_lost, right_lost - - with open(track_info_path, "r") as f: - track_info = json.load(f) - tracks = track_info.get("tracks", {}) - meta = track_info.get("meta", {}) - seq_interval = meta.get("seq_interval") # [start, end) in original frame indices - if seq_interval is not None and len(seq_interval) >= 2: - seq_start, seq_end = int(seq_interval[0]), int(seq_interval[1]) - else: - seq_start, seq_end = 0, None # legacy: use vis_mask from index 0 - - # Map track index (from world_results) to left/right using stats - for b, s in enumerate(stats): - # Find vis_mask for this track: track_info has track_id -> {index, vis_mask}; index matches our b - vis_mask = None - for info in tracks.values(): - if info.get("index") == b: - vis_mask = info.get("vis_mask") - break - if vis_mask is None: - continue - L = len(vis_mask) - if seq_end is not None: - # Slice vis_mask by seq_interval: NPZ frame t -> vis_mask[seq_start + t] - start = seq_start - end = min(seq_start + num_frames, L) - n = end - start - lost = [not bool(vis_mask[start + t]) for t in range(n)] - if n < num_frames: - lost += [True] * (num_frames - n) - else: - # Legacy: first num_frames of vis_mask - n = min(L, num_frames) - lost = [not bool(vis_mask[t]) for t in range(n)] - if n < num_frames: - lost += [True] * (num_frames - n) - if s.is_right is False: - left_lost = lost - elif s.is_right is True: - right_lost = lost - return left_lost, right_lost - - -def _jump_interpolate_mask(mask: List[bool], window: int = 10) -> List[bool]: - """ - For a jump mask (true=normal, false=jump): if frame t is a jump and any of the - following `window` frames is also a jump at t', set all frames from t to t' - (inclusive) to false in the result. Fills short gaps between nearby jumps. - """ - T = len(mask) - out = list(mask) - for t in range(T): - if mask[t]: - continue - for k in range(1, min(window + 1, T - t)): - if not mask[t + k]: - for i in range(t, t + k + 1): - out[i] = False - break - return out - - -def _filter_single_false(mask: List[bool], window: int = 10) -> List[bool]: - """ - Return a copy of mask where a single false is turned true if the previous - and following `window` frames are all true (isolated single-frame false). - """ - T = len(mask) - out = list(mask) - for t in range(T): - if mask[t]: - continue - prev_ok = t >= window and all(mask[i] for i in range(t - window, t)) - next_ok = t + window < T and all(mask[i] for i in range(t + 1, t + window + 1)) - if prev_ok and next_ok: - out[t] = True - return out - - -def build_per_frame_labels( - stats: List[TrackJumpStats], - num_frames: int, - left_detection_lost: List[bool], - right_detection_lost: List[bool], -) -> Dict[str, List[bool]]: - """Build per-frame mask arrays: true = normal/detected, false = jump/lost.""" - left_trans: set = set() - right_trans: set = set() - left_rot: set = set() - right_rot: set = set() - for s in stats: - # False = left, True = right; when is_right is None, assume track 0 = left, track 1 = right - is_left = (s.is_right is False) or (s.is_right is None and s.track_index == 0) - if is_left: - left_trans.update(s.trans_jump_frames) - left_rot.update(s.rot_jump_frames) - else: - right_trans.update(s.trans_jump_frames) - right_rot.update(s.rot_jump_frames) - - # Masks: true = normal/detected, false = jump/lost - left_trans_mask = [t not in left_trans for t in range(num_frames)] - left_rot_mask = [t not in left_rot for t in range(num_frames)] - right_trans_mask = [t not in right_trans for t in range(num_frames)] - right_rot_mask = [t not in right_rot for t in range(num_frames)] - left_trans_interp = _jump_interpolate_mask(left_trans_mask) - left_rot_interp = _jump_interpolate_mask(left_rot_mask) - right_trans_interp = _jump_interpolate_mask(right_trans_mask) - right_rot_interp = _jump_interpolate_mask(right_rot_mask) - left_det_mask = [not lost for lost in left_detection_lost] - right_det_mask = [not lost for lost in right_detection_lost] - - all_masks = ( - left_trans_mask, - left_rot_mask, - right_trans_mask, - right_rot_mask, - left_trans_interp, - left_rot_interp, - right_trans_interp, - right_rot_interp, - left_det_mask, - right_det_mask, - ) - valid_mask = [all(m[t] for m in all_masks) for t in range(num_frames)] - filtered_valid_mask = _filter_single_false(valid_mask) - - return { - "left_hand_translation_mask": left_trans_mask, - "left_hand_rotation_mask": left_rot_mask, - "right_hand_translation_mask": right_trans_mask, - "right_hand_rotation_mask": right_rot_mask, - "left_hand_translation_jump_interpolate_mask": left_trans_interp, - "left_hand_rotation_jump_interpolate_mask": left_rot_interp, - "right_hand_translation_jump_interpolate_mask": right_trans_interp, - "right_hand_rotation_jump_interpolate_mask": right_rot_interp, - "left_hand_detection_mask": left_det_mask, - "right_hand_detection_mask": right_det_mask, - "valid_mask": valid_mask, - "filtered_valid_mask": filtered_valid_mask, - } - - -def _summary_frames_false_by_key( - per_frame: Dict[str, List[bool]], -) -> Dict[str, List[int]]: - """For each mask key, list frame indices where the value is false.""" - return { - key: [t for t, v in enumerate(arr) if not v] for key, arr in per_frame.items() - } - - -def _threshold_info_from_stats( - stats: List[TrackJumpStats], -) -> Dict[str, Optional[float]]: - """Extract max trans_thr and max rot_thr_deg across tracks for batch median/error check.""" - trans_vals = [s.trans_thr for s in stats if s.trans_thr is not None] - rot_vals = [np.degrees(s.rot_thr) for s in stats if s.rot_thr is not None] - return { - "trans_thr": max(trans_vals) if trans_vals else None, - "rot_thr_deg": max(rot_vals) if rot_vals else None, - } - - -def process_one( - world_results: Path, - out_dir: Path, - args: argparse.Namespace, -) -> Optional[Tuple[Dict[str, List[bool]], Dict[str, Optional[float]]]]: - """Run jumpy-hand detection for a single world_results NPZ. Returns (per_frame, threshold_info) or None if no tracks.""" - d = np.load(world_results, allow_pickle=True) - required = ["trans", "root_orient"] - for k in required: - if k not in d: - raise KeyError( - f"Missing key '{k}' in {world_results}. Found keys: {list(d.keys())}" - ) - - trans = d["trans"] # (B,T,3) - root_orient = d["root_orient"] # (B,T,3) axis-angle - is_right = d["is_right"] if "is_right" in d else None # (B,T) or None - - stats, best = detect_jumpy_tracks( - trans_bt3=trans, - root_orient_bt3=root_orient, - is_right_bt=is_right, - trans_stat=args.trans_stat, - trans_factor=float(args.trans_factor), - trans_abs_threshold=args.trans_abs_threshold, - rot_stat=args.rot_stat, - rot_factor=float(args.rot_factor), - rot_abs_threshold_deg=args.rot_abs_threshold_deg, - ) - - if len(stats) == 0 or best < 0: - print(f"[{world_results}] No tracks found.") - return None - threshold_info = _threshold_info_from_stats(stats) - - print(f"Loaded: {world_results}") - print(f"Tracks (B)={trans.shape[0]}, Frames (T)={trans.shape[1]}") - print() - - for s in stats: - side = None - if s.is_right is not None: - side = "right" if s.is_right else "left" - trans_thr_str = "None" if s.trans_thr is None else f"{s.trans_thr:.6g}" - rot_thr_deg_str = ( - "None" if s.rot_thr is None else f"{np.degrees(s.rot_thr):.3f}°" - ) - print( - f"- track {s.track_index}" - + (f" ({side})" if side is not None else "") - + f": score={s.score:.3f} | " - + f"trans jumps={s.trans_jump_count} thr={trans_thr_str} max={s.trans_max if s.trans_max is not None else 'None'} | " - + f"rot jumps={s.rot_jump_count} thr={rot_thr_deg_str} max={s.rot_max_deg if s.rot_max_deg is not None else 'None'}°" - ) - - chosen = stats[best] - side = None if chosen.is_right is None else ("right" if chosen.is_right else "left") - print() - print( - f"Selected jumpy track: {chosen.track_index}" + (f" ({side})" if side else "") - ) - print(f" trans_jump_frames (first 30): {chosen.trans_jump_frames[:30]}") - print(f" rot_jump_frames (first 30): {chosen.rot_jump_frames[:30]}") - - # Optional outputs - report: Dict = { - "world_results": str(world_results), - "selected_track_index": int(chosen.track_index), - "selected_is_right": None if chosen.is_right is None else bool(chosen.is_right), - "tracks": [ - { - "track_index": int(s.track_index), - "is_right": None if s.is_right is None else bool(s.is_right), - "score": float(s.score), - "translation": { - "base": None if s.trans_base is None else float(s.trans_base), - "threshold": None if s.trans_thr is None else float(s.trans_thr), - "jump_count": int(s.trans_jump_count), - "jump_frames": [int(i) for i in s.trans_jump_frames], - "max": None if s.trans_max is None else float(s.trans_max), - }, - "rotation": { - "base_rad": None if s.rot_base is None else float(s.rot_base), - "threshold_rad": None if s.rot_thr is None else float(s.rot_thr), - "threshold_deg": None - if s.rot_thr is None - else float(np.degrees(s.rot_thr)), - "jump_count": int(s.rot_jump_count), - "jump_frames": [int(i) for i in s.rot_jump_frames], - "max_rad": None if s.rot_max_rad is None else float(s.rot_max_rad), - "max_deg": None if s.rot_max_deg is None else float(s.rot_max_deg), - }, - } - for s in stats - ], - "params": { - "trans_stat": args.trans_stat, - "trans_factor": float(args.trans_factor), - "trans_abs_threshold": args.trans_abs_threshold, - "rot_stat": args.rot_stat, - "rot_factor": float(args.rot_factor), - "rot_abs_threshold_deg": args.rot_abs_threshold_deg, - }, - } - - if args.json: - json_path = out_dir / "qc_world_jumpy_hand.json" - with open(json_path, "w") as f: - json.dump(report, f, indent=2) - print(f"Saved JSON: {json_path}") - - # Per-frame labels (always written when we have stats) - T = trans.shape[1] - track_info_path = ( - Path(args.track_info).expanduser().resolve() - if args.track_info - else world_results.parent.parent / "track_info.json" - ) - left_detection_lost, right_detection_lost = _load_track_info_detection_lost( - track_info_path, T, stats - ) - per_frame = build_per_frame_labels( - stats, T, left_detection_lost, right_detection_lost - ) - labels_path = out_dir / "qc_world_jumpy_per_frame_labels.json" - with open(labels_path, "w") as f: - json.dump(per_frame, f, indent=2) - print(f"Saved per-frame labels: {labels_path}") - - if args.plot: - dtrans_bt = adjacent_translation_delta(trans) - drot_bt = adjacent_rotation_angle_delta(root_orient) - _maybe_plot(out_dir, dtrans_bt, drot_bt, stats) - - return (per_frame, threshold_info) - - -def main() -> None: - ap = argparse.ArgumentParser() - ap.add_argument( - "--world_results", - type=str, - default=None, - help="Path to a single Dyn-HaMR `*_world_results.npz` (required if --world_results_dir not set).", - ) - ap.add_argument( - "--world_results_dir", - type=str, - default=None, - help="Directory to search for logs/video-custom/.../...-all-shot-0-0--1/smooth_fit/*_000300_world_results.npz; processes each (mutually exclusive with single file).", - ) - ap.add_argument( - "--out_subdir", - type=str, - default="qc_results", - help="When using --world_results_dir, output for each NPZ is written to .parent.parent / out_subdir (default: qc_results).", - ) - ap.add_argument( - "--out_dir", - type=str, - default=None, - help="Output directory for single-file mode (default: alongside the NPZ).", - ) - ap.add_argument( - "--track_info", - type=str, - default=None, - help="Path to track_info.json (default: .parent.parent / track_info.json).", - ) - ap.add_argument("--json", action="store_true", help="If set, write a JSON report.") - ap.add_argument( - "--plot", action="store_true", help="If set, write a PNG plot of deltas." - ) - - ap.add_argument( - "--trans_stat", type=str, default="median", choices=["median", "mean"] - ) - ap.add_argument( - "--trans_factor", - type=float, - default=20.0, - help="Effective trans thr = stat(dtrans) * factor.", - ) - ap.add_argument( - "--trans_abs_threshold", - type=float, - default=None, - help="Absolute translation jump threshold (overrides stat*factor). Units are the same as `trans` in NPZ.", - ) - - ap.add_argument( - "--rot_stat", type=str, default="median", choices=["median", "mean"] - ) - ap.add_argument( - "--rot_factor", - type=float, - default=20.0, - help="Effective rot thr = stat(drot_rad) * factor.", - ) - ap.add_argument( - "--rot_abs_threshold_deg", - type=float, - default=None, - help="Absolute rotation jump threshold in degrees (overrides stat*factor).", - ) - ap.add_argument( - "--error_threshold_factor_trans", - type=float, - default=10.0, - help="In batch mode: flag video as error_video if trans_thr > median_trans * this factor (default 10.0).", - ) - ap.add_argument( - "--error_threshold_factor_rot", - type=float, - default=10.0, - help="In batch mode: flag video as error_video if rot_thr_deg > median_rot_deg * this factor (default 10.0).", - ) - ap.add_argument( - "--error_valid_ratio", - type=float, - default=0.3, - help="In batch mode: also flag as error_video if filtered_invalid_ratio > this (default 0.3, i.e. >30%% invalid frames).", - ) - - args = ap.parse_args() - - if args.world_results_dir is not None: - # Batch mode: only logs/video-custom/xxx/xxx-all-shot-0-0--1/smooth_fit/xx_000300_world_results.npz - if args.world_results is not None: - ap.error("Cannot use both --world_results and --world_results_dir.") - base = Path(args.world_results_dir).expanduser().resolve() - if not base.is_dir(): - raise FileNotFoundError(f"Not a directory: {base}") - - def _is_video_custom_world_results(p: Path) -> bool: - if not p.is_file() or p.suffix != ".npz": - return False - if not p.name.endswith("_000300_world_results.npz"): - return False - if p.parent.name != "smooth_fit": - return False - if not p.parent.parent.name.endswith("-all-shot-0-0--1"): - return False - parts = p.parts - if "logs" not in parts or "video-custom" not in parts: - return False - return True - - npz_files = sorted( - p - for p in base.rglob("*_000300_world_results.npz") - if _is_video_custom_world_results(p) - ) - if not npz_files: - print( - f"No matching world_results found under {base} " - "(expect logs/video-custom/.../...-all-shot-0-0--1/smooth_fit/*_000300_world_results.npz)" - ) - return - print(f"Found {len(npz_files)} matching world_results NPZ file(s).") - batch_summary: List[Dict] = [] - for world_results in npz_files: - out_dir = world_results.parent.parent / args.out_subdir - out_dir.mkdir(parents=True, exist_ok=True) - print(f"\n{'=' * 60}\nProcessing: {world_results}") - try: - result = process_one(world_results, out_dir, args) - if result is not None: - per_frame, threshold_info = result - num_frames = len(next(iter(per_frame.values()))) - filtered_valid = per_frame.get("filtered_valid_mask") - if filtered_valid is not None and num_frames > 0: - invalid_count = sum(1 for x in filtered_valid if not x) - filtered_invalid_ratio = invalid_count / num_frames - else: - filtered_invalid_ratio = 0.0 - try: - rel = world_results.relative_to(base) - video_id = ( - rel.parts[0] - if rel.parts - else world_results.parent.parent.name - ) - except ValueError: - video_id = world_results.parent.parent.name - batch_summary.append( - { - "world_results": str(world_results), - "video_id": video_id, - "num_frames": num_frames, - "frames_false_by_key": _summary_frames_false_by_key( - per_frame - ), - "trans_thr": threshold_info["trans_thr"], - "rot_thr_deg": threshold_info["rot_thr_deg"], - "filtered_invalid_ratio": filtered_invalid_ratio, - } - ) - except Exception as e: - print(f"[ERROR] {world_results}: {e}") - raise - # Median thresholds and error_video flags - trans_vals = [ - v["trans_thr"] for v in batch_summary if v.get("trans_thr") is not None - ] - rot_vals = [ - v["rot_thr_deg"] for v in batch_summary if v.get("rot_thr_deg") is not None - ] - median_trans = float(np.median(trans_vals)) if trans_vals else None - median_rot_deg = float(np.median(rot_vals)) if rot_vals else None - factor_trans = float(args.error_threshold_factor_trans) - factor_rot = float(args.error_threshold_factor_rot) - error_valid_ratio_thr = float(args.error_valid_ratio) - error_video_ids: List[str] = [] - for v in batch_summary: - is_error = False - if median_trans is not None and v.get("trans_thr") is not None: - if v["trans_thr"] > median_trans * factor_trans: - is_error = True - if median_rot_deg is not None and v.get("rot_thr_deg") is not None: - if v["rot_thr_deg"] > median_rot_deg * factor_rot: - is_error = True - if v.get("filtered_invalid_ratio", 0.0) > error_valid_ratio_thr: - is_error = True - v["error_video"] = is_error - if is_error: - error_video_ids.append(v["video_id"]) - # Write batch summary - if batch_summary: - summary_path = base / "qc_world_jumpy_batch_summary.json" - summary_data = { - "world_results_dir": str(base), - "num_videos": len(batch_summary), - "median_trans_threshold": median_trans, - "median_rot_threshold_deg": median_rot_deg, - "error_threshold_factor_trans": factor_trans, - "error_threshold_factor_rot": factor_rot, - "error_valid_ratio": error_valid_ratio_thr, - "error_video_ids": error_video_ids, - "videos": batch_summary, - } - with open(summary_path, "w") as f: - json.dump(summary_data, f, indent=2) - print(f"\nSaved batch summary: {summary_path}") - if error_video_ids: - print( - f"Error videos (trans_thr > median*{factor_trans} or rot_thr_deg > median*{factor_rot} or filtered_invalid_ratio > {error_valid_ratio_thr}): {error_video_ids}" - ) - else: - print("No error videos (all within threshold and valid-ratio limits).") - return - - # Single-file mode - if args.world_results is None: - ap.error("Either --world_results or --world_results_dir is required.") - world_results = Path(args.world_results).expanduser().resolve() - if not world_results.is_file(): - raise FileNotFoundError(world_results) - out_dir = ( - Path(args.out_dir).expanduser().resolve() - if args.out_dir - else world_results.parent - ) - out_dir.mkdir(parents=True, exist_ok=True) - process_one(world_results, out_dir, args) - - -if __name__ == "__main__": - main() diff --git a/src/postprocessing/egocentric_hand_reconstruction/quality_control/run_jumpy_qc.sh b/src/postprocessing/egocentric_hand_reconstruction/quality_control/run_jumpy_qc.sh deleted file mode 100755 index 66ff51c9c2..0000000000 --- a/src/postprocessing/egocentric_hand_reconstruction/quality_control/run_jumpy_qc.sh +++ /dev/null @@ -1,69 +0,0 @@ -#!/bin/bash -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -# -# Run jumpy-hand QC scripts on Dyn-HaMR world_results (after scripts/run_reconstruction.sh). -# Uses host Python; no Docker required. See README_jumpy_qc.md for details. - -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)" -cd "${PROJECT_ROOT}" - -# Default: search under outputs/ for logs/.../video-custom/.../smooth_fit/*_world_results.npz -RESULTS_DIR="${1:-outputs}" -TRANS_FACTOR="${TRANS_FACTOR:-20.0}" -ROT_FACTOR="${ROT_FACTOR:-20.0}" -# FPS: optional second arg, or env FPS (for useful seconds in cal_jumpy_proportions) -FPS="${2:-${FPS:-}}" - -if [[ ! -d "${RESULTS_DIR}" ]]; then - echo "Error: Directory not found: ${RESULTS_DIR}" - echo "Usage: $0 [results_dir] [fps]" - echo " results_dir Directory containing Dyn-HaMR logs (default: outputs)" - echo " fps Optional FPS for useful seconds in proportions (e.g. 30)" - echo "" - echo "Optional env: TRANS_FACTOR, ROT_FACTOR (default 20.0), FPS (if not passed as second arg)." - exit 1 -fi - -RESULTS_DIR="$(cd "${RESULTS_DIR}" && pwd)" -echo "==========================================" -echo "Jumpy-hand QC (host Python)" -echo "==========================================" -echo "Results dir: ${RESULTS_DIR}" -echo "" - -# Step 1: Detect jumpy hands (writes qc_results per run + qc_world_jumpy_batch_summary.json in results_dir) -echo "Step 1: Running detect_jumpy_hand_from_world_results.py ..." -python3 "${SCRIPT_DIR}/detect_jumpy_hand_from_world_results.py" \ - --world_results_dir "${RESULTS_DIR}" \ - --json \ - --plot \ - --trans_factor "${TRANS_FACTOR}" \ - --rot_factor "${ROT_FACTOR}" \ - --error_threshold_factor_trans 10.0 \ - --error_threshold_factor_rot 10.0 \ - --error_valid_ratio 0.3 - -SUMMARY="${RESULTS_DIR}/qc_world_jumpy_batch_summary.json" -if [[ ! -f "${SUMMARY}" ]]; then - echo "No batch summary produced (no matching world_results found?). Skipping proportions." - exit 0 -fi - -# Step 2: Compute proportions from batch summary -echo "" -echo "Step 2: Running cal_jumpy_proportions.py ..." -EXTRA="" -[[ -n "${FPS}" ]] && EXTRA="${EXTRA} --fps ${FPS}" -python3 "${SCRIPT_DIR}/cal_jumpy_proportions.py" --world_results_dir "${RESULTS_DIR}" ${EXTRA} - -echo "" -echo "==========================================" -echo "Jumpy-hand QC done." -echo "==========================================" -echo " Batch summary: ${SUMMARY}" -echo " Per-run QC: /qc_results/" -echo " Proportions: printed above; use --out in cal_jumpy_proportions.py to save JSON." diff --git a/src/postprocessing/egocentric_hand_reconstruction/scripts/download_video_from_bucket.py b/src/postprocessing/egocentric_hand_reconstruction/scripts/download_video_from_bucket.py deleted file mode 100644 index f7ccd0a887..0000000000 --- a/src/postprocessing/egocentric_hand_reconstruction/scripts/download_video_from_bucket.py +++ /dev/null @@ -1,113 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -import os -import sys -from urllib.parse import urlparse - -try: - import boto3 - from botocore.config import Config -except Exception as exc: # pragma: no cover - print("Error: boto3 is required for bucket downloads.") - print(f"Install with: pip install boto3 (details: {exc})") - sys.exit(1) - - -def parse_bucket_url(video_url: str) -> tuple[str | None, str, str]: - """Parse a bucket URL into (host, bucket, key). - - Supported schemes: - s3:/// - swift://// - swift:///AUTH_// - - Returns (host, bucket, key) where host is None for s3:// URLs. - """ - parsed = urlparse(video_url) - - if parsed.scheme == "s3": - bucket = parsed.netloc - key = parsed.path.lstrip("/") - if not bucket or not key: - raise ValueError( - "s3:// URL must include bucket and key. Expected: s3:///" - ) - return None, bucket, key - - if parsed.scheme == "swift": - host = parsed.netloc - parts = [p for p in parsed.path.split("/") if p] - if len(parts) < 2: - raise ValueError( - "swift:// URL must include bucket and key. " - "Expected: swift://// or " - "swift:///AUTH_//" - ) - if parts[0].startswith("AUTH_"): - if len(parts) < 3: - raise ValueError( - "swift:// URL with AUTH_ must include bucket and key" - ) - bucket = parts[1] - key = "/".join(parts[2:]) - else: - bucket = parts[0] - key = "/".join(parts[1:]) - return host, bucket, key - - raise ValueError(f"unsupported URI scheme '{parsed.scheme}' in {video_url}") - - -def main() -> int: - if len(sys.argv) != 3: - print( - "Usage: python3 download_video_from_bucket.py " - ) - print(" Supported schemes: s3://, swift://") - return 1 - - video_url = sys.argv[1] - destination_path = sys.argv[2] - - try: - host, bucket, key = parse_bucket_url(video_url) - except ValueError as exc: - print(f"Error: {exc}") - return 1 - - parsed = urlparse(video_url) - - access_key_id = os.environ.get("ACCESS_KEY_ID") - secret_access_key = os.environ.get("SECRET_ACCESS_KEY") - region = os.environ.get("BUCKET_REGION", "us-east-1") - endpoint_url = os.environ.get("BUCKET_ENDPOINT_URL") - - if not endpoint_url and parsed.scheme == "swift" and host: - endpoint_url = f"https://{host}" - - if not access_key_id or not secret_access_key: - print("Error: missing credentials.") - print("Set ACCESS_KEY_ID and SECRET_ACCESS_KEY.") - return 1 - - client_kwargs = dict( - aws_access_key_id=access_key_id, - aws_secret_access_key=secret_access_key, - region_name=region, - config=Config(connect_timeout=5), - ) - if endpoint_url: - client_kwargs["endpoint_url"] = endpoint_url - - os.makedirs(os.path.dirname(destination_path), exist_ok=True) - - s3 = boto3.client("s3", **client_kwargs) - s3.download_file(bucket, key, destination_path) - - print(f"Downloaded {video_url} -> {destination_path}") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/src/postprocessing/egocentric_hand_reconstruction/scripts/run_mesh_generation.sh b/src/postprocessing/egocentric_hand_reconstruction/scripts/run_mesh_generation.sh deleted file mode 100755 index ce58c57a4b..0000000000 --- a/src/postprocessing/egocentric_hand_reconstruction/scripts/run_mesh_generation.sh +++ /dev/null @@ -1,55 +0,0 @@ -#!/bin/bash -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -# -# Host-side wrapper that runs mesh generation inside the Dyn-HaMR container. -# Expects Dyn-HaMR reconstruction results under OUTPUTS_DIR/logs/...//*_world_results.npz. - -set -euo pipefail - -PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -OUTPUTS_DIR="${OUTPUTS_DIR:-${PROJECT_ROOT}/outputs}" - -PHASE="${PHASE:-smooth_fit}" -GPU="${GPU:-0}" -NO_TEMPORAL_SMOOTH="${NO_TEMPORAL_SMOOTH:-0}" -NO_SMOOTH_TRANS="${NO_SMOOTH_TRANS:-0}" - -if [[ ! -d "${OUTPUTS_DIR}" ]]; then - echo "Error: outputs directory not found: ${OUTPUTS_DIR}" - echo "Run scripts/run_reconstruction.sh first, or set OUTPUTS_DIR." - exit 1 -fi - -# MANO_RIGHT.pkl lives on the host in OUTPUTS_DIR/; setup_dynhamr.sh copies it -# into the container's _DATA/data/mano/ on first run. Gate here for a clear error. -if [[ ! -f "${OUTPUTS_DIR}/MANO_RIGHT.pkl" ]]; then - echo "Error: MANO_RIGHT.pkl not found in ${OUTPUTS_DIR}/" - echo "Download from https://mano.is.tue.mpg.de/ and place at ${OUTPUTS_DIR}/MANO_RIGHT.pkl" - exit 1 -fi - -echo "==========================================" -echo "Hand mesh generation (via Dyn-HaMR container)" -echo "==========================================" -echo "Outputs dir: ${OUTPUTS_DIR}" -echo "Phase: ${PHASE}" -echo "GPU: ${GPU}" -echo "" - -OUTPUTS_DIR="${OUTPUTS_DIR}" "${PROJECT_ROOT}/docker/dynhamr.sh" run bash -c " - # Ensure MANO is populated inside the container (idempotent). - bash /home/appuser/setup_dynhamr.sh >/dev/null - - PHASE='${PHASE}' \ - GPU='${GPU}' \ - NO_TEMPORAL_SMOOTH='${NO_TEMPORAL_SMOOTH}' \ - NO_SMOOTH_TRANS='${NO_SMOOTH_TRANS}' \ - bash /home/appuser/Dyn-HaMR/dyn-hamr/generate_mesh.sh /home/appuser/outputs -" - -echo "" -echo "==========================================" -echo "Mesh generation complete." -echo "==========================================" -echo "Hand mesh .npz files written alongside world_results under ${OUTPUTS_DIR}/logs/" diff --git a/src/postprocessing/egocentric_hand_reconstruction/scripts/run_reconstruction.sh b/src/postprocessing/egocentric_hand_reconstruction/scripts/run_reconstruction.sh deleted file mode 100755 index 58764a7fa8..0000000000 --- a/src/postprocessing/egocentric_hand_reconstruction/scripts/run_reconstruction.sh +++ /dev/null @@ -1,133 +0,0 @@ -#!/bin/bash -# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -set -euo pipefail - -# Check arguments -if [[ $# -lt 1 ]]; then - echo "Usage: $0 " - echo "Example: $0 my-local-path/my-video.mp4" - echo "Example: $0 s3://my-remote-path/my-video.mp4" - echo "" - echo "For bucket URLs (s3://), set credentials in environment variables:" - echo " ACCESS_KEY_ID" - echo " SECRET_ACCESS_KEY" - echo "Optional:" - echo " BUCKET_REGION (default: us-east-1)" - echo " BUCKET_ENDPOINT_URL" - exit 1 -fi - -# Configuration -PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -OUTPUTS_DIR="${OUTPUTS_DIR:-${PROJECT_ROOT}/outputs}" -VIDEO_PATH="$1" -VIDEO_FULL_PATH="" -VIDEO_FILE="" -BUCKET_DOWNLOADER="${PROJECT_ROOT}/scripts/download_video_from_bucket.py" - -mkdir -p "${OUTPUTS_DIR}" - -is_bucket_url() { - [[ "$1" == swift://* || "$1" == s3://* ]] -} - -if is_bucket_url "${VIDEO_PATH}"; then - VIDEO_FILE="$(basename "${VIDEO_PATH}")" - VIDEO_FULL_PATH="${OUTPUTS_DIR}/${VIDEO_FILE}" -else - if [[ -f "${VIDEO_PATH}" ]]; then - VIDEO_FULL_PATH="${VIDEO_PATH}" - else - VIDEO_FULL_PATH="${PROJECT_ROOT}/${VIDEO_PATH}" - fi - if [[ ! -f "${VIDEO_FULL_PATH}" ]]; then - echo "Error: Video file not found at ${VIDEO_FULL_PATH}" - exit 1 - fi - VIDEO_FILE="$(basename "${VIDEO_FULL_PATH}")" -fi - -# Extract video name (without extension) -VIDEO_NAME="${VIDEO_FILE%.*}" - -# Check for required licensed data (needed by Dyn-HaMR in Step 3) -missing=0 -if [[ ! -f "${OUTPUTS_DIR}/MANO_RIGHT.pkl" ]]; then - echo "Error: MANO_RIGHT.pkl not found in ${OUTPUTS_DIR}/" - missing=1 -fi -if [[ ! -d "${OUTPUTS_DIR}/BMC" ]] || [[ -z "$(ls -A "${OUTPUTS_DIR}/BMC/" 2>/dev/null)" ]]; then - echo "Error: BMC data not found in ${OUTPUTS_DIR}/BMC/" - missing=1 -fi -if [[ "$missing" -eq 1 ]]; then - exit 1 -fi - -echo "Processing video: ${VIDEO_PATH}" -echo "Video name: ${VIDEO_NAME}" - -echo "" -echo "==========================================" -echo "Step 1: Preparing video for processing" -echo "==========================================" - -if is_bucket_url "${VIDEO_PATH}"; then - if [[ ! -f "${VIDEO_FULL_PATH}" ]]; then - if ! command -v python3 >/dev/null 2>&1; then - echo "Error: python3 is required to download bucket URLs." - exit 1 - fi - if [[ ! -f "${BUCKET_DOWNLOADER}" ]]; then - echo "Error: Bucket downloader script not found at ${BUCKET_DOWNLOADER}" - exit 1 - fi - python3 "${BUCKET_DOWNLOADER}" "${VIDEO_PATH}" "${VIDEO_FULL_PATH}" - echo "✓ Video downloaded to outputs directory" - else - echo "✓ Video already downloaded in outputs directory" - fi -else - # Copy local video to outputs if not already there - if [[ ! -f "${OUTPUTS_DIR}/${VIDEO_FILE}" ]]; then - cp "${VIDEO_FULL_PATH}" "${OUTPUTS_DIR}/${VIDEO_FILE}" - echo "✓ Video copied to outputs directory" - else - echo "✓ Video already in outputs directory" - fi -fi - -echo "" -echo "==========================================" -echo "Step 2: Running ViPE for camera estimation" -echo "==========================================" -"${PROJECT_ROOT}/docker/vipe.sh" run vipe infer "vipe_results/${VIDEO_FILE}" - -echo "" -echo "==========================================" -echo "Step 3: Running Dyn-HaMR (setup + reconstruction)" -echo "==========================================" -"${PROJECT_ROOT}/docker/dynhamr.sh" run bash -c " - # Run setup script (configs, dependencies, HMP disable) - bash /home/appuser/setup_dynhamr.sh - - # Create symlink to video inside container - ln -sf /home/appuser/outputs/${VIDEO_FILE} /home/appuser/Dyn-HaMR/test/videos/${VIDEO_FILE} - echo '✓ Video linked inside container' - - # Run reconstruction - cd /home/appuser/Dyn-HaMR/dyn-hamr - python run_opt.py \ - data=video_vipe \ - run_opt=True \ - run_vis=True \ - data.seq=${VIDEO_NAME} \ - is_static=False -" - -echo "" -echo "==========================================" -echo "Pipeline Complete!" -echo "==========================================" -echo "Results saved to: ${OUTPUTS_DIR}/logs/"