Merge branch 'main' into feature/icpflow

Author: Kin-Zhang

.gitignore

@@ -25,4 +25,7 @@ cancel.sh
 # cuda build files
 *.egg-info
 build*
-dist*
+dist*
+
+data
+*__pycache__

CONTRIBUTING.md

@@ -1,10 +1,10 @@
-# Contributing to OpenSceneFlow
+# Contributing to [OpenSceneFlow](https://github.com/KTH-RPL/OpenSceneFlow)
 
 We want to make contributing to this project as easy and transparent as possible. We welcome any contributions, from bug fixes to new features. If you're interested in adding your own scene flow method, this guide will walk you through the process.
 
 ## Adding a New Method
 
-Here is a quick guide to integrating a new method into the OpenSceneFlow codebase.
+Here is a quick guide to integrating a new method into the [OpenSceneFlow](https://github.com/KTH-RPL/OpenSceneFlow) codebase.
 
 ### 1. Data Preparation
 

Dockerfile

@@ -1,8 +1,15 @@
-FROM pytorch/pytorch:2.1.0-cuda11.8-cudnn8-devel
+# check more: https://hub.docker.com/r/nvidia/cuda
+FROM nvidia/cuda:11.7.1-devel-ubuntu20.04
 ENV DEBIAN_FRONTEND noninteractive
 LABEL maintainer="Qingwen Zhang <https://kin-zhang.github.io/>"
 
-RUN apt update && apt install -y git tmux curl vim rsync libgl1 libglib2.0-0 ca-certificates
+RUN apt update && apt install -y git curl vim rsync htop
+
+RUN curl -o ~/miniforge3.sh -LO https://github.com/conda-forge/miniforge/releases/latest/download/miniforge3-Linux-x86_64.sh  && \
+    chmod +x ~/miniforge3.sh && \
+    ~/miniforge3.sh -b -p /opt/conda && \
+    rm ~/miniforge3.sh && \
+    /opt/conda/bin/conda clean -ya && /opt/conda/bin/conda init bash && /opt/conda/bin/conda init zsh
 
 # install zsh and oh-my-zsh
 RUN apt update && apt install -y wget git zsh tmux vim g++
@@ -14,23 +21,28 @@ RUN sh -c "$(wget -O- https://github.com/deluan/zsh-in-docker/releases/download/
     -p https://github.com/zsh-users/zsh-syntax-highlighting
 
 RUN printf "y\ny\ny\n\n" | bash -c "$(curl -fsSL https://raw.githubusercontent.com/Kin-Zhang/Kin-Zhang/main/scripts/setup_ohmyzsh.sh)"
-RUN /opt/conda/bin/conda init zsh
 
 # change to conda env
 ENV PATH /opt/conda/bin:$PATH
-RUN /opt/conda/bin/conda config --set solver libmamba
 
-RUN mkdir -p /home/kin/workspace && cd /home/kin/workspace && git clone https://github.com/Kin-Zhang/OpenSceneFlow
+RUN mkdir -p /home/kin/workspace && cd /home/kin/workspace && git clone https://github.com/KTH-RPL/OpenSceneFlow.git
 WORKDIR /home/kin/workspace/OpenSceneFlow
+RUN apt-get update && apt-get install libgl1 -y
 
 # need read the gpu device info to compile the cuda extension
-RUN /opt/conda/bin/pip install -r /home/kin/workspace/OpenSceneFlow/requirements.txt
-RUN /opt/conda/bin/pip install FastGeodis --no-build-isolation
-RUN /opt/conda/bin/pip install --no-cache-dir -e ./assets/cuda/chamfer3D && /opt/conda/bin/pip install --no-cache-dir -e ./assets/cuda/mmcv
+RUN cd /home/kin/workspace/OpenSceneFlow && /opt/conda/bin/conda env create -f environment.yaml
+# To make images can run all methods in the codebase
+RUN /opt/conda/envs/opensf/bin/pip install torch-scatter -f https://data.pyg.org/whl/torch-2.0.0+cu117.html
+RUN /opt/conda/envs/opensf/bin/pip install FastGeodis --no-build-isolation --no-cache-dir
+RUN /opt/conda/envs/opensf/bin/pip install mmengine-lite && \
+    /opt/conda/bin/conda install -n opensf -y pytorch3d -c pytorch3d
+
+# custom cuda library
+RUN cd /home/kin/workspace/OpenSceneFlow/assets/cuda/mmcv && /opt/conda/envs/opensf/bin/python ./setup.py install
+RUN cd /home/kin/workspace/OpenSceneFlow/assets/cuda/chamfer3D && /opt/conda/envs/opensf/bin/python ./setup.py install
 
-# environment for dataprocessing includes data-api
-RUN /opt/conda/bin/conda env create -f envsftool.yaml
+RUN cd /home/kin/workspace/OpenSceneFlow && /opt/conda/bin/conda env create -f envsftool.yaml
 RUN /opt/conda/envs/sftool/bin/pip install numpy==1.22
 
 # clean up apt cache
-RUN rm -rf /var/lib/apt/lists/* && rm -rf /root/.cache/pip
+RUN rm -rf /var/lib/apt/lists/* && rm -rf /root/.cache/pip && /opt/conda/bin/conda clean -ya

README.md

@@ -11,10 +11,20 @@
 OpenSceneFlow is a codebase for point cloud scene flow estimation. 
 It is also an official implementation of the following papers (sorted by the time of publication):
 
+- **DeltaFlow: An Efficient Multi-frame Scene Flow Estimation Method**   
+*Qingwen Zhang, Xiaomeng Zhu, Yushan Zhang, Yixi Cai, Olov Andersson, Patric Jensfelt*  
+Preprint; Under review; 2025   
+[ Backbone ] [ Supervised ] - [ [arXiv](https://arxiv.org/abs/2508.17054) ] [ [Project](https://github.com/Kin-Zhang/DeltaFlow) ]
+
 - **HiMo: High-Speed Objects Motion Compensation in Point Clouds** (SeFlow++)   
 *Qingwen Zhang, Ajinkya Khoche, Yi Yang, Li Ling, Sina Sharif Mansouri, Olov Andersson, Patric Jensfelt*  
-Preprint; Under review; 2025   
-[ Strategy ] [ Self-Supervised ] - [ [arXiv](https://arxiv.org/abs/2503.00803) ] [ [Project](https://kin-zhang.github.io/HiMo/) ]
+IEEE Transactions on Robotics (**T-RO**) 2025   
+[ Strategy ] [ Self-Supervised ] - [ [arXiv](https://arxiv.org/abs/2503.00803) ] [ [Project](https://kin-zhang.github.io/HiMo/) ] → [here](#seflow-1)
+
+- **VoteFlow: Enforcing Local Rigidity in Self-Supervised Scene Flow**   
+*Yancong Lin\*, Shiming Wang\*, Liangliang Nan, Julian Kooij, Holger Caesar*   
+Conference on Computer Vision and Pattern Recognition (**CVPR**) 2025  
+[ Backbone ] [ Self-Supervised ] - [ [arXiv](https://arxiv.org/abs/2503.22328) ] [ [Project](https://github.com/tudelft-iv/VoteFlow/) ] → [here](#VoteFLow)
 
 - **Flow4D: Leveraging 4D Voxel Network for LiDAR Scene Flow Estimation**  
 *Jaeyeul Kim, Jungwan Woo, Ukcheol Shin, Jean Oh, Sunghoon Im*  
@@ -50,21 +60,12 @@ Additionally, *OpenSceneFlow* integrates following excellent works: [ICLR'24 Zer
 - [x] [ZeroFlow](https://arxiv.org/abs/2305.10424): ICLR 2024, their pre-trained weight can covert into our format easily through [the script](tools/zerof2ours.py).
 - [x] [NSFP](https://arxiv.org/abs/2111.01253): NeurIPS 2021, faster 3x than original version because of [our CUDA speed up](assets/cuda/README.md), same (slightly better) performance.
 - [x] [FastNSF](https://arxiv.org/abs/2304.09121): ICCV 2023. SSL optimization-based.
+- [ ] [EulerFlow](https://arxiv.org/abs/2410.02031): ICLR 2025. SSL optimization-based. In my plan, haven't coding yet.
 
 </details>
 
 💡: Want to learn how to add your own network in this structure? Check [Contribute section](CONTRIBUTING.md#adding-a-new-method) and know more about the code. Fee free to pull request and your bibtex [here](#cite-us).
 
----
-
-<!-- 📜 Changelog:
-
-- 🎁 2025/1/28 14:58: Update the codebase to collect all methods in one repository reference [Pointcept](https://github.com/Pointcept/Pointcept) repo.
-- 🤗 2024/11/18 16:17: Update model and demo data download link through HuggingFace, Personally I found `wget` from HuggingFace link is much faster than Zenodo.
-- 2024/09/26 16:24: All codes already uploaded and tested. You can to try training directly by downloading (through [HuggingFace](https://huggingface.co/kin-zhang/OpenSceneFlow)/[Zenodo](https://zenodo.org/records/13744999)) demo data or pretrained weight for evaluation. 
-- 2024/07/24: Merging SeFlow & DeFlow code together, lighter setup and easier running.
-- 🔥 2024/07/02: Check the self-supervised version in our new ECCV'24 [SeFlow](https://github.com/KTH-RPL/SeFlow). The 1st ranking in new leaderboard among self-supervise methods. -->
-
 ## 0. Installation
 
 There are two ways to install the codebase: directly on your [local machine](#environment-setup) or in a [Docker container](#docker-recommended-for-isolation).
@@ -75,7 +76,7 @@ We use conda to manage the environment, you can install it follow [here](assets/
 
 ```bash
 git clone --recursive https://github.com/KTH-RPL/OpenSceneFlow.git
-cd OpenSceneFlow && mamba env create -f environment.yaml
+cd OpenSceneFlow && conda env create -f environment.yaml
 
 # You may need export your LD_LIBRARY_PATH with env lib
 # export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/kin/mambaforge/lib
@@ -98,23 +99,24 @@ cd /home/kin/workspace/OpenSceneFlow && git pull
 cd /home/kin/workspace/OpenSceneFlow/assets/cuda/mmcv && /opt/conda/envs/opensf/bin/python ./setup.py install
 cd /home/kin/workspace/OpenSceneFlow/assets/cuda/chamfer3D && /opt/conda/envs/opensf/bin/python ./setup.py install
 cd /home/kin/workspace/OpenSceneFlow
-mamba activate opensf
+conda activate opensf
 ```
 
 If you prefer to build the Docker image by yourself, Check [build-docker-image](assets/README.md#build-docker-image) section for more details.
 
 ## 1. Data Preparation
 
-Refer to [dataprocess/README.md](dataprocess/README.md) for dataset download instructions. Currently, we support **Argoverse 2**, **Waymo**, and **custom datasets** (more datasets will be added in the future). 
+Refer to [dataprocess/README.md](dataprocess/README.md) for dataset download instructions. Currently, we support **Argoverse 2**, **Waymo**, **nuScenes**, **ZOD** and **custom datasets** (more datasets will be added in the future). 
 
 After downloading, convert the raw data to `.h5` format for easy training, evaluation, and visualization. Follow the steps in [dataprocess/README.md#process](dataprocess/README.md#process). 
 
 For a quick start, use our **mini processed dataset**, which includes one scene in `train` and `val`. It is pre-converted to `.h5` format with label data ([HuggingFace](https://huggingface.co/kin-zhang/OpenSceneFlow/blob/main/demo_data.zip)/[Zenodo](https://zenodo.org/records/13744999/files/demo_data.zip)).
 
 
 ```bash
-wget https://huggingface.co/kin-zhang/OpenSceneFlow/resolve/main/demo_data.zip
-unzip demo_data.zip -d /home/kin/data/av2/h5py
+# around 1.3G
+wget https://huggingface.co/kin-zhang/OpenSceneFlow/resolve/main/demo-data-v2.zip
+unzip demo-data-v2.zip -d /home/kin/data/av2/h5py
 ```
 
 Once extracted, you can directly use this dataset to run the [training script](#2-quick-start) without further processing.
@@ -129,35 +131,34 @@ Some tips before running the code:
 And free yourself from trainning, you can download the pretrained weight from [HuggingFace](https://huggingface.co/kin-zhang/OpenSceneFlow) and we provided the detail `wget` command in each model section. For optimization-based method, it's train-free so you can directly run with [3. Evaluation](#3-evaluation) (check more in the evaluation section).
 
 ```bash
-mamba activate opensf
+conda activate opensf
 ```
 
-### Flow4D
+### Supervised Training
+
+#### Flow4D
 
 Train Flow4D with the leaderboard submit config. [Runtime: Around 18 hours in 4x RTX 3090 GPUs.]
 
 ```bash
-python train.py model=flow4d lr=1e-3 epochs=15 batch_size=8 num_frames=5 loss_fn=deflowLoss "voxel_size=[0.2, 0.2, 0.2]" "point_cloud_range=[-51.2, -51.2, -3.2, 51.2, 51.2, 3.2]"
-```
+python train.py model=flow4d optimizer.lr=1e-3 epochs=15 batch_size=8 num_frames=5 loss_fn=deflowLoss "voxel_size=[0.2, 0.2, 0.2]" "point_cloud_range=[-51.2, -51.2, -3.2, 51.2, 51.2, 3.2]"
 
-Pretrained weight can be downloaded through:
-```bash
+# Pretrained weight can be downloaded through:
 wget https://huggingface.co/kin-zhang/OpenSceneFlow/resolve/main/flow4d_best.ckpt
 ```
 
-### SSF
+#### SSF
 
 Extra packages needed for SSF model:
 ```bash
-pip install mmengine-lite torch-scatter
-# torch-scatter might not working, then reinstall by:
-pip install https://data.pyg.org/whl/torch-2.0.0%2Bcu118/torch_scatter-2.1.2%2Bpt20cu118-cp310-cp310-linux_x86_64.whl
+pip install mmengine-lite
+pip install torch-scatter -f https://data.pyg.org/whl/torch-2.0.0+cu117.html
 ```
 
 Train SSF with the leaderboard submit config. [Runtime: Around 6 hours in 8x A100 GPUs.]
 
 ```bash
-python train.py model=ssf lr=8e-3 epochs=25 batch_size=64 loss_fn=deflowLoss "voxel_size=[0.2, 0.2, 6]" "point_cloud_range=[-51.2, -51.2, -3, 51.2, 51.2, 3]"
+python train.py model=ssf optimizer.lr=8e-3 epochs=25 batch_size=64 loss_fn=deflowLoss "voxel_size=[0.2, 0.2, 6]" "point_cloud_range=[-51.2, -51.2, -3, 51.2, 51.2, 3]"
 ```
 
 Pretrained weight can be downloaded through:
@@ -169,48 +170,82 @@ wget https://huggingface.co/kin-zhang/OpenSceneFlow/resolve/main/ssf_best.ckpt
 wget https://huggingface.co/kin-zhang/OpenSceneFlow/resolve/main/ssf_long.ckpt
 ```
 
+#### DeFlow
 
-### SeFlow
-
-Train SeFlow needed to specify the loss function, we set the config of our best model in the leaderboard. [Runtime: Around 11 hours in 4x A100 GPUs.]
+Train DeFlow with the leaderboard submit config. [Runtime: Around 6-8 hours in 4x A100 GPUs.] Please change `batch_size&lr` accoordingly if you don't have enough GPU memory. (e.g. `batch_size=6` for 24GB GPU)
 
 ```bash
-python train.py model=deflow lr=2e-4 epochs=9 batch_size=16 loss_fn=seflowLoss "add_seloss={chamfer_dis: 1.0, static_flow_loss: 1.0, dynamic_chamfer_dis: 1.0, cluster_based_pc0pc1: 1.0}" "model.target.num_iters=2"
+python train.py model=deflow optimizer.lr=2e-4 epochs=15 batch_size=16 loss_fn=deflowLoss
+
+# Pretrained weight can be downloaded through:
+wget https://huggingface.co/kin-zhang/OpenSceneFlow/resolve/main/deflow_best.ckpt
 ```
 
-Pretrained weight can be downloaded through:
+### Feed-Forward Self-Supervised Model Training
+
+Train Feed-forward SSL methods (e.g. SeFlow/SeFlow++/VoteFlow etc), we needed to:
+1) process auto-label process.
+2) specify the loss function, we set the config here for our best model in the leaderboard.
+
+#### SeFlow
+
 ```bash
+# [Runtime: Around 11 hours in 4x A100 GPUs.]
+python train.py model=deflow optimizer.lr=2e-4 epochs=9 batch_size=16 loss_fn=seflowLoss +ssl_label=seflow_auto "+add_seloss={chamfer_dis: 1.0, static_flow_loss: 1.0, dynamic_chamfer_dis: 1.0, cluster_based_pc0pc1: 1.0}" "model.target.num_iters=2"
+
+# Pretrained weight can be downloaded through:
 wget https://huggingface.co/kin-zhang/OpenSceneFlow/resolve/main/seflow_best.ckpt
 ```
 
-### ICP-Flow
+#### VoteFLow
+Extra pakcges needed for VoteFlow, [pytorch3d](https://pytorch3d.org/) (prefer 0.7.7) and [torch-scatter](https://github.com/rusty1s/pytorch_scatter?tab=readme-ov-file) (prefer 2.1.2):
 
-ICP-Flow is a optimization-based method, you can directly run `eval.py`/`save.py` to get the result. 
+```bash
+# Install Pytorch3d
+conda install pytorch3d -c pytorch3d
 
-Extra packages needed for ICP-Flow model:
+# Install torch-scatter
+pip install torch-scatter -f https://data.pyg.org/whl/torch-2.0.0+cu117.html
+```
+
+Train VoteFlow with the leaderboard submit config. [Runtime: Around 32 hours in 4 x V100 GPUs.]
 ```bash
-pip install pytorch3d assets/cuda/histlib
+python train.py model=voteflow optimizer.lr=2e-4 +optimizer.scheduler.name=StepLR +optimizer.scheduler.step_size=6 epochs=12 batch_size=4 model.target.m=8 model.target.n=128 loss_fn=seflowLoss "+add_seloss={chamfer_dis: 1.0, static_flow_loss: 1.0, dynamic_chamfer_dis: 1.0, cluster_based_pc0pc1: 1.0}" +ssl_label=seflow_auto
+
+# Pretrained weight can be downloaded through:
+wget https://huggingface.co/kin-zhang/OpenSceneFlow/resolve/main/voteflow_best.ckpt
 ```
 
-Then run as:
+
+#### SeFlow++
+
 ```bash
-python eval.py model=icpflow
-python save.py model=icpflow
+# [Runtime: Around 10 hours in 4x A100 GPUs.] for Argoverse 2
+python train.py model=deflowpp save_top_model=3 val_every=3 voxel_size="[0.2, 0.2, 6]" point_cloud_range="[-51.2, -51.2, -3, 51.2, 51.2, 3]" num_workers=16 epochs=9 optimizer.lr=2e-4 +optimizer.scheduler.name=StepLR "+add_seloss={chamfer_dis: 1.0, static_flow_loss: 1.0, dynamic_chamfer_dis: 1.0, cluster_based_pc0pc1: 1.0}" +ssl_label=seflowpp_auto loss_fn=seflowppLoss num_frames=3 batch_size=4
+
+# Pretrained weight can be downloaded through:
+wget https://huggingface.co/kin-zhang/OpenSceneFlow/resolve/main/seflowpp_best.ckpt
 ```
 
-### DeFlow
 
-Train DeFlow with the leaderboard submit config. [Runtime: Around 6-8 hours in 4x A100 GPUs.] Please change `batch_size&lr` accoordingly if you don't have enough GPU memory. (e.g. `batch_size=6` for 24GB GPU)
+### Optimization-based Unsupervised Methods
+
+For all optimization-based methods, you can directly run `eval.py`/`save.py` to get the result without training, while the running might take really long time, maybe tmux for run it.
+
+#### ICP-Flow
 
+Extra packages needed for ICP-Flow model:
 ```bash
-python train.py model=deflow lr=2e-4 epochs=15 batch_size=16 loss_fn=deflowLoss
+pip install pytorch3d assets/cuda/histlib
 ```
 
-Pretrained weight can be downloaded through:
+Then run as:
 ```bash
-wget https://huggingface.co/kin-zhang/OpenSceneFlow/resolve/main/deflow_best.ckpt
+python eval.py model=icpflow
+python save.py model=icpflow
 ```
 
+
 ## 3. Evaluation
 
 You can view Wandb dashboard for the training and evaluation results or upload result to online leaderboard.
@@ -219,14 +254,14 @@ Since in training, we save all hyper-parameters and model checkpoints, the only
 
 ```bash
 # (feed-forward): load ckpt and run it, it will directly prints all metric
-python eval.py checkpoint=/home/kin/seflow_best.ckpt av2_mode=val
+python eval.py checkpoint=/home/kin/seflow_best.ckpt data_mode=val
 
 # (optimization-based): it might need take really long time, maybe tmux for run it.
 python eval.py model=nsfp
 
 # it will output the av2_submit.zip or av2_submit_v2.zip for you to submit to leaderboard
-python eval.py checkpoint=/home/kin/seflow_best.ckpt av2_mode=test leaderboard_version=1
-python eval.py checkpoint=/home/kin/seflow_best.ckpt av2_mode=test leaderboard_version=2
+python eval.py checkpoint=/home/kin/seflow_best.ckpt data_mode=test leaderboard_version=1
+python eval.py checkpoint=/home/kin/seflow_best.ckpt data_mode=test leaderboard_version=2
 ```
 
 ### **📊 Range-Wise Metric (New!)**
@@ -243,13 +278,13 @@ In [SSF paper](https://arxiv.org/abs/2501.17821), we introduce a new distance-ba
 
 
 ### Submit result to public leaderboard
-To submit your result to the public Leaderboard, if you select `av2_mode=test`, it should be a zip file for you to submit to the leaderboard.
+To submit your result to the public Leaderboard, if you select `data_mode=test`, it should be a zip file for you to submit to the leaderboard.
 Note: The leaderboard result in DeFlow&SeFlow main paper is [version 1](https://eval.ai/web/challenges/challenge-page/2010/evaluation), as [version 2](https://eval.ai/web/challenges/challenge-page/2210/overview) is updated after DeFlow&SeFlow.
 
 ```bash
 # since the env may conflict we set new on deflow, we directly create new one:
-mamba create -n py37 python=3.7
-mamba activate py37
+conda create -n py37 python=3.7
+conda activate py37
 pip install "evalai"
 
 # Step 2: login in eval and register your team
@@ -346,6 +381,13 @@ And our excellent collaborators works contributed to this codebase also:
   journal={arXiv preprint arXiv:2501.17821},
   year={2025}
 }
+
+@inproceedings{lin2025voteflow,
+  title={VoteFlow: Enforcing Local Rigidity in Self-Supervised Scene Flow},
+  author={Lin, Yancong and Wang, Shiming and Nan, Liangliang and Kooij, Julian and Caesar, Holger},
+  booktitle={CVPR},
+  year={2025},
+}
 @article{lin2024icp,
   title={ICP-Flow: LiDAR Scene Flow Estimation with ICP},
   author={Lin, Yancong and Caesar, Holger},

assets/README.md

@@ -80,6 +80,7 @@ python -c "import torch; print(torch.__version__); print(torch.cuda.is_available
 python -c "import lightning.pytorch as pl; print(pl.__version__)"
 python -c "from assets.cuda.mmcv import Voxelization, DynamicScatter;print('successfully import on our lite mmcv package')"
 python -c "from assets.cuda.chamfer3D import nnChamferDis;print('successfully import on our chamfer3D package')"
+python -c "from av2.utils.io import read_feather; print('av2 package ok')"
 ```
 
 
@@ -100,3 +101,6 @@ python -c "from assets.cuda.chamfer3D import nnChamferDis;print('successfully im
 
 3. torch_scatter problem: `OSError: /home/kin/mambaforge/envs/opensf-v2/lib/python3.10/site-packages/torch_scatter/_version_cpu.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE`
    Solved by install the torch-cuda version: `pip install https://data.pyg.org/whl/torch-2.0.0%2Bcu118/torch_scatter-2.1.2%2Bpt20cu118-cp310-cp310-linux_x86_64.whl`
+
+4. cuda package problem: `ValueError(f"Unknown CUDA arch ({arch}) or GPU not supported")`
+   Solved by [checking GPU compute](https://developer.nvidia.cn/cuda-gpus#compute) then manually assign: `export TORCH_CUDA_ARCH_LIST=8.6`