Skip to content

Commit 596fe1c

Browse files
alejoe91alejoe91
committed
IBL version of pyKS and simplify running tests locally
2 parents cebc9d9 + 664db73 commit 596fe1c

33 files changed

Lines changed: 522 additions & 855 deletions

‎hdsort-compiled/Dockerfile‎

Lines changed: 15 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -3,14 +3,19 @@ FROM hdsort-matlab-base
33
RUN chmod 755 /usr/bin/mlrtapp/hdsort_compiled
44
ENV PATH="/usr/bin/mlrtapp:${PATH}"
55

6-
RUN apt-get update -y
7-
RUN apt-get install software-properties-common -y
8-
RUN add-apt-repository ppa:deadsnakes/ppa -y
9-
RUN apt-get install git python3.8 python3.8-dev -y
10-
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 1
11-
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 2
12-
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10
13-
RUN apt-get install python3-pip -y
14-
RUN update-alternatives --install /usr/bin/pip pip /usr/bin/pip3 1
15-
RUN pip install -U pip
6+
# Installing Python with miniconda
7+
RUN apt-get update && \
8+
apt-get install -y build-essential && \
9+
apt-get install -y wget && \
10+
apt-get install -y git && \
11+
apt-get clean && \
12+
rm -rf /var/lib/apt/lists/*
1613

14+
ENV MINICONDA_VERSION 4.8.2
15+
ENV CONDA_DIR /home/miniconda3
16+
ENV LATEST_CONDA_SCRIPT "Miniconda3-py38_$MINICONDA_VERSION-Linux-x86_64.sh"
17+
18+
RUN wget --quiet https://repo.anaconda.com/miniconda/$LATEST_CONDA_SCRIPT -O ~/miniconda.sh && \
19+
bash ~/miniconda.sh -b -p $CONDA_DIR && \
20+
rm ~/miniconda.sh
21+
ENV PATH=$CONDA_DIR/bin:$PATH

‎hdsort-compiled/README.md‎

Lines changed: 12 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -1,60 +1,37 @@
1-
# HDsort Compiled Docker Images
1+
# HDsort Compiled Docker Image
22

33
This documentation is intended to show how to create a Docker image with Matlab compiled HDsort sorter. The main goal of this project is to avoid the requirement of Matlab Licenses and also abstract the installation and setup steps to run the sorter
44

55
There are three main steps for generating a functional HDsort docker image:
66
1. Compile HDsort as Matlab's Standalone Application
7-
2. Create a (base) docker image with Matlab Runtime and the compiled application from step 1
7+
2. Create a base docker image with Matlab Runtime and the compiled application from step 1
88
3. Extend the docker image from step 2 for improvements and fixes
99

10+
Steps 1 and 2 are done by `compile.sh` script, while step 3 is done by `build.sh` script
11+
1012
## Requirements
1113
- Packaging a MATLAB Docker image is supported on Linux only
1214
- Docker
1315
- Matlab
1416

1517
### Matlab Requirements
1618
- MATLAB Compiler
19+
- Fuzzy_Logic Toolbox
1720
- More requirements to be checked and listed...
1821

19-
## Compiling HDsort as Matlab's Standalone Application
22+
## Cloning repositories
2023
- Git clone or Download HDsort [source code](https://git.bsse.ethz.ch/hima_public/HDsort)
21-
- Open Matlab
22-
- Set Matlab's workspace folder to `/path/to/spikeinterface-dockerfiles/hdsort-compiled`
23-
- Run `mcc` command with `hdsort_compiled.m` as base and adding path to git cloned HDSort:
24-
```matlab
25-
>> mcc -m hdsort_compiled.m -a <git-cloned-path>/HDsort
26-
```
24+
- Git clone or Download spikeinterface [source code](https://github.com/SpikeInterface/spikeinterface)
2725

2826
## Generating Base Docker Image
29-
- To generate the base docker image (called `hdsort-matlab-base`) with the compiled application, run the following command in Matlab console:
30-
```matlab
31-
>> compiler.package.docker('hdsort_compiled', 'requiredMCRProducts.txt', 'ImageName', 'hdsort-matlab-base')
27+
`compile.sh` script generates the base docker image (called `hdsort-matlab-base`) with the matlab compiled HDSort application
28+
- Run compile script:
29+
```bash
30+
$ source compile.sh /path/to/HDSort /path/to/spikeinterface
3231
```
3332

34-
35-
- [Optional] Files generated by Matlab Compiler can be deleted:
36-
- In your terminal, go to the `hdsort-compiled` folder in this project:
37-
```bash
38-
$ cd /path/to/spikeinterface-dockerfiles/hdsort-compiled
39-
```
40-
41-
- Run `rm` command:
42-
```bash
43-
$ rm -r \
44-
hdsort-matlab-basedocker \
45-
hdsort_compiled \
46-
includedSupportPackages.txt \
47-
mccExcludedFiles.log \
48-
readme.txt \
49-
requiredMCRProducts.txt \
50-
run_hdsort_compiled.sh \
51-
unresolvedSymbols.txt
52-
```
53-
54-
5533
## Extending Base Image and creating final image
56-
The Dockerfile in this folder applies some fixes and updates to the base image generated automatically by Matlab in order to properly run waveclus:
57-
34+
The Dockerfile in this folder applies some fixes and updates to the base image in order to properly run HDSort:
5835

5936
- In your terminal, go to the folder for this project:
6037
```

‎hdsort-compiled/compile.sh‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
#!/bin/bash
2+
set -e
3+
4+
if [ $# == 0 ]; then
5+
echo "Usage: $0 param1 param2"
6+
echo "* param1: hdsort path"
7+
echo "* param2: spikeinterface path"
8+
exit
9+
fi
10+
11+
if [ $# -ne 2 ]; then
12+
echo "spikeinterface and hdsort path must be given"
13+
exit 1
14+
fi
15+
16+
HDSORT_COMPILED_NAME="hdsort_compiled"
17+
HDSORT_PATH=${1%/}
18+
SI_PATH=${2%/}
19+
WORK_DIR=$(pwd)
20+
SOURCE_DIR=$( dirname -- "$0"; )
21+
TMP_DIR=$SOURCE_DIR/tmp
22+
23+
echo "hdsort path: ${HDSORT_PATH}"
24+
echo "spike-interface path: ${SI_PATH}"
25+
26+
echo "Creating tmp folder: $TMP_DIR"
27+
cd $WORK_DIR
28+
mkdir -p $TMP_DIR
29+
30+
echo "Compiling hdsort..."
31+
cd $TMP_DIR
32+
matlab -batch "mcc -m ${SI_PATH}/spikeinterface/sorters/hdsort/hdsort_master.m -a ${SI_PATH}/spikeinterface/sorters/utils -a ${HDSORT_PATH} -o ${HDSORT_COMPILED_NAME}"
33+
34+
echo "Creating base docker image..."
35+
matlab -batch "compiler.package.docker('${HDSORT_COMPILED_NAME}', 'requiredMCRProducts.txt', 'ImageName', 'hdsort-matlab-base')"
36+
37+
cd $WORK_DIR
38+
rm -r $TMP_DIR

‎hdsort-compiled/hdsort_compiled.m‎

Lines changed: 0 additions & 57 deletions
This file was deleted.

‎ironclust-compiled/Dockerfile‎

Lines changed: 15 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -3,14 +3,19 @@ FROM ironclust-matlab-base
33
RUN chmod 755 /usr/bin/mlrtapp/p_ironclust
44
ENV PATH="/usr/bin/mlrtapp:${PATH}"
55

6-
RUN apt-get update -y
7-
RUN apt-get install software-properties-common -y
8-
RUN add-apt-repository ppa:deadsnakes/ppa -y
9-
RUN apt-get install git python3.8 python3.8-dev -y
10-
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 1
11-
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 2
12-
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10
13-
RUN apt-get install python3-pip -y
14-
RUN update-alternatives --install /usr/bin/pip pip /usr/bin/pip3 1
15-
RUN pip install -U pip
6+
# Installing Python with miniconda
7+
RUN apt-get update && \
8+
apt-get install -y build-essential && \
9+
apt-get install -y wget && \
10+
apt-get install -y git && \
11+
apt-get clean && \
12+
rm -rf /var/lib/apt/lists/*
1613

14+
ENV MINICONDA_VERSION 4.8.2
15+
ENV CONDA_DIR /home/miniconda3
16+
ENV LATEST_CONDA_SCRIPT "Miniconda3-py38_$MINICONDA_VERSION-Linux-x86_64.sh"
17+
18+
RUN wget --quiet https://repo.anaconda.com/miniconda/$LATEST_CONDA_SCRIPT -O ~/miniconda.sh && \
19+
bash ~/miniconda.sh -b -p $CONDA_DIR && \
20+
rm ~/miniconda.sh
21+
ENV PATH=$CONDA_DIR/bin:$PATH

‎ironclust-compiled/README.md‎

Lines changed: 9 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ This documentation is intended to show how to create a Docker image with Matlab
44

55
There are three main steps for generating a functional ironclust docker image:
66
1. Compile Ironclust as Matlab's Standalone Application
7-
2. Create a (base) docker image with Matlab Runtime and the compiled application from step 1
7+
2. Create a base docker image with Matlab Runtime and the compiled application from step 1
88
3. Extend the docker image from step 2 for improvements and fixes
99

1010
## Requirements
@@ -21,42 +21,20 @@ There are three main steps for generating a functional ironclust docker image:
2121

2222
Licenses for Matlab and toolboxes are needed only for compiling ironclust as Standalone Application and to generate the base Docker image. After this process, no license will be required, either to extend the base image or to run the sorter.
2323

24-
## Compiling Ironclust as Matlab's Standalone Application
24+
## Cloning repositories
2525
- Git clone or Download ironclust [source code](https://github.com/flatironinstitute/ironclust.git)
26-
- Open Matlab
27-
- Set Matlab's workspace folder to: `<git-cloned-path>/ironclust/matlab`
28-
- In Matlab console run:
29-
```
30-
>> mcc -m p_ironclust.m -a <git-cloned-path>/ironclust/matlab/*.* -a <git-cloned-path>/ironclust/matlab/prb -a <git-cloned-path>/ironclust/matlab/prb_json
31-
```
26+
- Git clone or Download spikeinterface [source code](https://github.com/SpikeInterface/spikeinterface)
27+
3228

3329
## Generating Base Docker Image
34-
- To generate the base docker image (called `ironclust-matlab-base`) with the compiled application, run the following command in Matlab console:
35-
```
36-
>> compiler.package.docker('p_ironclust', 'requiredMCRProducts.txt', 'ImageName', 'ironclust-matlab-base')
30+
`compile.sh` script generates the base docker image (called `ironclust-matlab-base`) with the matlab compiled ironclust application
31+
- Run compile script:
32+
```bash
33+
$ source compile.sh /path/to/ironclust /path/to/spikeinterface
3734
```
3835

39-
- [Optional] Files generated by Matlab Compiler can be deleted:
40-
- In your terminal, go to the folder for this project:
41-
```
42-
$ cd <git-cloned-path>/ironclust/
43-
```
44-
- Run `rm` command:
45-
```
46-
$ rm -r \
47-
matlab/includedSupportPackages.txt \
48-
matlab/ironclust-matlab-basedocker \
49-
matlab/mccExcludedFiles.log \
50-
matlab/p_ironclust \
51-
matlab/readme.txt \
52-
matlab/requiredMCRProducts.txt \
53-
matlab/run_p_ironclust.sh \
54-
matlab/unresolvedSymbols.txt
55-
```
56-
57-
5836
## Extending Base Image/Creating final image
59-
The Dockerfile in this folder applies some fixes and updates to the base image generated automatically by Matlab in order to properly run ironclust:
37+
The Dockerfile in this folder applies some fixes and updates to the base image in order to properly run ironclust:
6038

6139
- In your terminal, go to the folder for this project:
6240
```
@@ -68,7 +46,6 @@ $ cd /path/to/spikeinterface-dockerfiles/ironclust-compiled
6846
$ source build.sh
6947
```
7048

71-
7249
## Running a container
7350

7451
- [nvidia-container-toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#setting-up-nvidia-container-toolkit) is required to run a docker with GPU capabilities

‎ironclust-compiled/compile.sh‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
#!/bin/bash
2+
set -e
3+
4+
if [ $# == 0 ]; then
5+
echo "Usage: $0 param1"
6+
echo "* param1: Ironclust path"
7+
fi
8+
9+
if [ $# -ne 1 ]; then
10+
echo "ironclust path must be given"
11+
exit 1
12+
fi
13+
14+
IC_COMPILED_NAME="p_ironclust"
15+
IC_PATH=${1%/}/matlab
16+
WORK_DIR=$(pwd)
17+
SOURCE_DIR=$( dirname -- "$0"; )
18+
TMP_DIR=$SOURCE_DIR/tmp
19+
20+
echo "Ironclust path: ${IC_PATH}"
21+
22+
echo "Compiling CUDA files"
23+
cd $IC_PATH
24+
matlab -batch "irc2 compile"
25+
26+
echo "Creating tmp folder: ${TMP_DIR}"
27+
cd $WORK_DIR
28+
mkdir -p $TMP_DIR
29+
30+
echo "Compiling p_ironclust..."
31+
cd $TMP_DIR
32+
33+
# Generating multiple "-a filename" string
34+
# This is needed because wildcard pattern /* doesn't work
35+
# properly when running mcc outside matlab's console
36+
ADD_FILES=""
37+
for fname in $(eval "ls ${IC_PATH} -I \"*.pdf\" -p | grep -v /"); do
38+
ADD_FILES="${ADD_FILES} -a ${IC_PATH}/${fname}"
39+
done
40+
ADD_FILES="${ADD_FILES} -a ${IC_PATH}/prb"
41+
ADD_FILES="${ADD_FILES} -a ${IC_PATH}/prb_json"
42+
43+
matlab -batch "mcc -m ${IC_PATH}/p_ironclust.m ${ADD_FILES} -o ${IC_COMPILED_NAME}"
44+
45+
echo "Creating base docker image..."
46+
matlab -batch "compiler.package.docker('${IC_COMPILED_NAME}', 'requiredMCRProducts.txt', 'ImageName', 'ironclust-matlab-base')"
47+
48+
cd $WORK_DIR
49+
rm -r $TMP_DIR

‎kilosort_no_license/README.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,12 +18,14 @@ This documentation is intended to show how to create a Docker image with Matlab
1818

1919
There are four main steps to generate a functional kilosort-compiled docker image:
2020

21-
1. Kilosort Setup
21+
1. Compile Kilosort CUDA files
2222
2. Compile Kilosort as Standalone Application
2323
3. Create a (base) docker image with Matlab Runtime and the compiled application from step 2
2424
4. Extend the docker image from step 3 for improvements and fixes
2525

26-
Detailed descriptions for each step are documented in `README.md` for each supported kilosort version folder (`kilosort-compiled`, `kilosort2-compiled` and so on)
26+
Steps 1 to 3 are done by `compile.sh` script, while step 4 is done by `build.sh` script.
27+
28+
Steps to run scripts and create the images are documented in `README.md` for each supported kilosort version folder (`kilosort-compiled`, `kilosort2-compiled` and so on)
2729

2830
Licenses for Matlab and toolboxes are needed only for compiling kilosort as Standalone Application and to generate the base Docker image. After this process, no license will be required, either to extend the base image or to run the sorter.
2931

‎kilosort_no_license/kilosort-compiled/Dockerfile‎

Lines changed: 15 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -3,14 +3,19 @@ FROM ks-matlab-base
33
RUN chmod 755 /usr/bin/mlrtapp/ks_compiled
44
ENV PATH="/usr/bin/mlrtapp:${PATH}"
55

6-
RUN apt-get update -y
7-
RUN apt-get install software-properties-common -y
8-
RUN add-apt-repository ppa:deadsnakes/ppa -y
9-
RUN apt-get install git python3.8 python3.8-dev -y
10-
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 1
11-
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 2
12-
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10
13-
RUN apt-get install python3-pip -y
14-
RUN update-alternatives --install /usr/bin/pip pip /usr/bin/pip3 1
15-
RUN pip install -U pip
6+
# Installing Python with miniconda
7+
RUN apt-get update && \
8+
apt-get install -y build-essential && \
9+
apt-get install -y wget && \
10+
apt-get install -y git && \
11+
apt-get clean && \
12+
rm -rf /var/lib/apt/lists/*
1613

14+
ENV MINICONDA_VERSION 4.8.2
15+
ENV CONDA_DIR /home/miniconda3
16+
ENV LATEST_CONDA_SCRIPT "Miniconda3-py38_$MINICONDA_VERSION-Linux-x86_64.sh"
17+
18+
RUN wget --quiet https://repo.anaconda.com/miniconda/$LATEST_CONDA_SCRIPT -O ~/miniconda.sh && \
19+
bash ~/miniconda.sh -b -p $CONDA_DIR && \
20+
rm ~/miniconda.sh
21+
ENV PATH=$CONDA_DIR/bin:$PATH

0 commit comments

Comments
 (0)