Skip to content

Commit e962b48

Browse files
callum-jpgbethac07
callum-jpg
and
bethac07
authored
Add headless pipeline test to github actions (#204)
* add initially working headless test action * change model type * test on PR too * split jobs * retest the test * split job test with steps * Split testing --------- Co-authored-by: bethac07 <bcimini@broadinstitute.org>
1 parent 136a3e0 commit e962b48

8 files changed

Lines changed: 507 additions & 0 deletions

‎.github/workflows/test.yml‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
name: Basic plugin testing
2+
3+
on:
4+
push:
5+
branches:
6+
- master
7+
pull_request:
8+
types: [opened, reopened, synchronize, ready_for_review, review_requested]
9+
10+
jobs:
11+
build_cellprofiler:
12+
name: Build CellProfiler
13+
runs-on: ubuntu-latest
14+
steps:
15+
- uses: actions/checkout@v2
16+
- uses: actions/setup-python@v2
17+
with:
18+
architecture: x64
19+
python-version: ${{ matrix.python-version }}
20+
- name: Get pip cache dir
21+
id: pip-cache-dir
22+
run: |
23+
echo "::set-output name=dir::$(pip cache dir)"
24+
- name: Cache pip
25+
uses: actions/cache@v3
26+
id: cache-pip
27+
with:
28+
path: ${{ steps.pip-cache-dir.outputs.dir }}
29+
key: ${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('**/setup.py') }}
30+
- name: Set up Python
31+
uses: actions/setup-python@v2
32+
with:
33+
python-version: 3.8
34+
35+
- name: Install CellProfiler dependencies
36+
run: |
37+
sudo apt-get update
38+
sudo apt install -y make gcc build-essential libgtk-3-dev
39+
sudo apt-get install -y python3-pip openjdk-11-jdk-headless default-libmysqlclient-dev libnotify-dev libsdl2-dev
40+
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
41+
export PATH=$PATH:/home/ubuntu/.local/bin
42+
43+
- name: Install CellProfiler
44+
run: |
45+
wget https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-20.04/wxPython-4.1.0-cp38-cp38-linux_x86_64.whl
46+
pip install wxPython-4.1.0-cp38-cp38-linux_x86_64.whl
47+
pip install --upgrade pip setuptools wheel
48+
pip install numpy
49+
50+
test_basic_plugins:
51+
needs: build_cellprofiler
52+
- name: Install basic CellProfiler plugins (require )
53+
run: |
54+
pip install -e .
55+
- name: Run basic plugin pipeline
56+
run: |
57+
python -m cellprofiler -c -r -p ./tests/headless_test/4.2.5_plugins_test_pipeline_BASIC.cppipe -i ./tests/headless_test/test_pipeline_img -o . --plugins-directory=./active_plugins --log-level=DEBUG 2>&1 | tee logfile
58+
sh ./tests/headless_test/test_run.sh ", module compensatecolors" logfile
59+
sh ./tests/headless_test/test_run.sh "module distancetransform" logfile
60+
sh ./tests/headless_test/test_run.sh "module enhancedmeasuretexture" logfile
61+
sh ./tests/headless_test/test_run.sh "module histogramequalization" logfile
62+
sh ./tests/headless_test/test_run.sh "module histogrammatching" logfile
63+
sh ./tests/headless_test/test_run.sh "module pixelshuffle" logfile
64+
65+
test_cellpose_plugin:
66+
needs: build_cellprofiler
67+
- name: Run cellpose plugin pipeline
68+
run: |
69+
pip install -e .[cellpose]
70+
python -m cellprofiler -c -r -p ./tests/headless_test/4.2.5_plugins_test_pipeline_CELLPOSE.cppipe -i ./tests/headless_test/test_pipeline_img -o . --plugins-directory=./active_plugins --log-level=DEBUG 2>&1 | tee logfile
71+
sh ./tests/headless_test/test_run.sh "module runcellpose" logfile
72+
73+
test_stardist_plugin:
74+
needs: build_cellprofiler
75+
- name: Run stardist plugin pipeline
76+
run: |
77+
pip uninstall -y cellpose torch
78+
pip install -e .[stardist]
79+
python -m cellprofiler -c -r -p ./tests/headless_test/4.2.5_plugins_test_pipeline_STARDIST.cppipe -i ./tests/headless_test/test_pipeline_img -o . --plugins-directory=./active_plugins --log-level=DEBUG 2>&1 | tee logfile
80+
sh ./tests/headless_test/test_run.sh "module runstardist" logfile

‎.github/workflows/test_cellpose.yml‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
name: CellProfiler-Cellpose
2+
3+
on:
4+
push:
5+
branches:
6+
- master
7+
pull_request:
8+
types: [opened, reopened, synchronize, ready_for_review, review_requested]
9+
10+
jobs:
11+
build_cellprofiler:
12+
name: Build CellProfiler
13+
runs-on: ubuntu-latest
14+
steps:
15+
- uses: actions/checkout@v2
16+
- uses: actions/setup-python@v2
17+
with:
18+
architecture: x64
19+
python-version: ${{ matrix.python-version }}
20+
- name: Get pip cache dir
21+
id: pip-cache-dir
22+
run: |
23+
echo "::set-output name=dir::$(pip cache dir)"
24+
- name: Cache pip
25+
uses: actions/cache@v3
26+
id: cache-pip
27+
with:
28+
path: ${{ steps.pip-cache-dir.outputs.dir }}
29+
key: ${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('**/setup.py') }}
30+
- name: Set up Python
31+
uses: actions/setup-python@v2
32+
with:
33+
python-version: 3.8
34+
35+
- name: Install CellProfiler dependencies
36+
run: |
37+
sudo apt-get update
38+
sudo apt install -y make gcc build-essential libgtk-3-dev
39+
sudo apt-get install -y python3-pip openjdk-11-jdk-headless default-libmysqlclient-dev libnotify-dev libsdl2-dev
40+
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
41+
export PATH=$PATH:/home/ubuntu/.local/bin
42+
43+
- name: Install CellProfiler
44+
run: |
45+
wget https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-20.04/wxPython-4.1.0-cp38-cp38-linux_x86_64.whl
46+
pip install wxPython-4.1.0-cp38-cp38-linux_x86_64.whl
47+
pip install --upgrade pip setuptools wheel
48+
pip install numpy
49+
50+
51+
test_cellpose_plugin:
52+
needs: build_cellprofiler
53+
- name: Run cellpose plugin pipeline
54+
run: |
55+
pip install -e .[cellpose]
56+
python -m cellprofiler -c -r -p ./tests/headless_test/4.2.5_plugins_test_pipeline_CELLPOSE.cppipe -i ./tests/headless_test/test_pipeline_img -o . --plugins-directory=./active_plugins --log-level=DEBUG 2>&1 | tee logfile
57+
sh ./tests/headless_test/test_run.sh "module runcellpose" logfile
58+

‎.github/workflows/test_stardist.yml‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
name: CellProfiler-Stardist
2+
3+
on:
4+
push:
5+
branches:
6+
- master
7+
pull_request:
8+
types: [opened, reopened, synchronize, ready_for_review, review_requested]
9+
10+
jobs:
11+
build_cellprofiler:
12+
name: Build CellProfiler
13+
runs-on: ubuntu-latest
14+
steps:
15+
- uses: actions/checkout@v2
16+
- uses: actions/setup-python@v2
17+
with:
18+
architecture: x64
19+
python-version: ${{ matrix.python-version }}
20+
- name: Get pip cache dir
21+
id: pip-cache-dir
22+
run: |
23+
echo "::set-output name=dir::$(pip cache dir)"
24+
- name: Cache pip
25+
uses: actions/cache@v3
26+
id: cache-pip
27+
with:
28+
path: ${{ steps.pip-cache-dir.outputs.dir }}
29+
key: ${{ runner.os }}-${{ env.pythonLocation }}-${{ hashFiles('**/setup.py') }}
30+
- name: Set up Python
31+
uses: actions/setup-python@v2
32+
with:
33+
python-version: 3.8
34+
35+
- name: Install CellProfiler dependencies
36+
run: |
37+
sudo apt-get update
38+
sudo apt install -y make gcc build-essential libgtk-3-dev
39+
sudo apt-get install -y python3-pip openjdk-11-jdk-headless default-libmysqlclient-dev libnotify-dev libsdl2-dev
40+
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
41+
export PATH=$PATH:/home/ubuntu/.local/bin
42+
43+
- name: Install CellProfiler
44+
run: |
45+
wget https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-20.04/wxPython-4.1.0-cp38-cp38-linux_x86_64.whl
46+
pip install wxPython-4.1.0-cp38-cp38-linux_x86_64.whl
47+
pip install --upgrade pip setuptools wheel
48+
pip install numpy
49+
50+
test_stardist_plugin:
51+
needs: build_cellprofiler
52+
- name: Run stardist plugin pipeline
53+
run: |
54+
pip uninstall -y cellpose torch
55+
pip install -e .[stardist]
56+
python -m cellprofiler -c -r -p ./tests/headless_test/4.2.5_plugins_test_pipeline_STARDIST.cppipe -i ./tests/headless_test/test_pipeline_img -o . --plugins-directory=./active_plugins --log-level=DEBUG 2>&1 | tee logfile
57+
sh ./tests/headless_test/test_run.sh "module runstardist" logfile

‎tests/headless_test/4.2.5_plugins_test_pipeline_BASIC.cppipe‎

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
CellProfiler Pipeline: http://www.cellprofiler.org
2+
Version:5
3+
DateRevision:425
4+
GitHash:
5+
ModuleCount:11
6+
HasImagePlaneDetails:False
7+
8+
Images:[module_num:1|svn_version:'Unknown'|variable_revision_number:2|show_window:True|notes:['To begin creating your project, use the Images module to compile a list of files and/or folders that you want to analyze. You can also specify a set of rules to include only the desired files in your selected folders.']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
9+
:
10+
Filter images?:Images only
11+
Select the rule criteria:and (extension does isimage) (directory doesnot containregexp "[\\\\/]\\.")
12+
13+
Metadata:[module_num:2|svn_version:'Unknown'|variable_revision_number:6|show_window:True|notes:['The Metadata module optionally allows you to extract information describing your images (i.e, metadata) which will be stored along with your measurements. This information can be contained in the file name and/or location, or in an external file.']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
14+
Extract metadata?:No
15+
Metadata data type:Text
16+
Metadata types:{}
17+
Extraction method count:1
18+
Metadata extraction method:Extract from file/folder names
19+
Metadata source:File name
20+
Regular expression to extract from file name:^(?P<Plate>.*)_(?P<Well>[A-P][0-9]{2})_s(?P<Site>[0-9])_w(?P<ChannelNumber>[0-9])
21+
Regular expression to extract from folder name:(?P<Date>[0-9]{4}_[0-9]{2}_[0-9]{2})$
22+
Extract metadata from:All images
23+
Select the filtering criteria:and (file does contain "")
24+
Metadata file location:Elsewhere...|
25+
Match file and image metadata:[]
26+
Use case insensitive matching?:No
27+
Metadata file name:None
28+
Does cached metadata exist?:No
29+
30+
NamesAndTypes:[module_num:3|svn_version:'Unknown'|variable_revision_number:8|show_window:True|notes:['The NamesAndTypes module allows you to assign a meaningful name to each image by which other modules will refer to it.']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
31+
Assign a name to:All images
32+
Select the image type:Grayscale image
33+
Name to assign these images:DNA
34+
Match metadata:[]
35+
Image set matching method:Order
36+
Set intensity range from:Image metadata
37+
Assignments count:1
38+
Single images count:0
39+
Maximum intensity:255.0
40+
Process as 3D?:No
41+
Relative pixel spacing in X:1.0
42+
Relative pixel spacing in Y:1.0
43+
Relative pixel spacing in Z:1.0
44+
Select the rule criteria:and (file does contain "")
45+
Name to assign these images:DNA
46+
Name to assign these objects:Cell
47+
Select the image type:Grayscale image
48+
Set intensity range from:Image metadata
49+
Maximum intensity:255.0
50+
51+
Groups:[module_num:4|svn_version:'Unknown'|variable_revision_number:2|show_window:True|notes:['The Groups module optionally allows you to split your list of images into image subsets (groups) which will be processed independently of each other. Examples of groupings include screening batches, microtiter plates, time-lapse movies, etc.']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
52+
Do you want to group your images?:No
53+
grouping metadata count:1
54+
Metadata category:None
55+
56+
IdentifyPrimaryObjects:[module_num:5|svn_version:'Unknown'|variable_revision_number:15|show_window:True|notes:[]|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
57+
Select the input image:DNA
58+
Name the primary objects to be identified:IdentifyPrimaryObjects
59+
Typical diameter of objects, in pixel units (Min,Max):10,40
60+
Discard objects outside the diameter range?:Yes
61+
Discard objects touching the border of the image?:Yes
62+
Method to distinguish clumped objects:Intensity
63+
Method to draw dividing lines between clumped objects:Intensity
64+
Size of smoothing filter:10
65+
Suppress local maxima that are closer than this minimum allowed distance:7.0
66+
Speed up by using lower-resolution image to find local maxima?:Yes
67+
Fill holes in identified objects?:After both thresholding and declumping
68+
Automatically calculate size of smoothing filter for declumping?:Yes
69+
Automatically calculate minimum allowed distance between local maxima?:Yes
70+
Handling of objects if excessive number of objects identified:Continue
71+
Maximum number of objects:500
72+
Use advanced settings?:No
73+
Threshold setting version:12
74+
Threshold strategy:Global
75+
Thresholding method:Minimum Cross-Entropy
76+
Threshold smoothing scale:1.3488
77+
Threshold correction factor:1.0
78+
Lower and upper bounds on threshold:0.0,1.0
79+
Manual threshold:0.0
80+
Select the measurement to threshold with:None
81+
Two-class or three-class thresholding?:Two classes
82+
Log transform before thresholding?:No
83+
Assign pixels in the middle intensity class to the foreground or the background?:Foreground
84+
Size of adaptive window:50
85+
Lower outlier fraction:0.05
86+
Upper outlier fraction:0.05
87+
Averaging method:Mean
88+
Variance method:Standard deviation
89+
# of deviations:2.0
90+
Thresholding method:Minimum Cross-Entropy
91+
92+
CompensateColors:[module_num:6|svn_version:'Unknown'|variable_revision_number:1|show_window:True|notes:[]|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
93+
Hidden:2
94+
Hidden:1
95+
Select an image to measure:DNA
96+
What compensation class does this image belong to?:1
97+
Select an output image name:None
98+
Select an image to measure:DNA
99+
What compensation class does this image belong to?:1
100+
Select an output image name:None
101+
Select where to perform color compensation:Across entire image
102+
Select an object to perform compensation within:None
103+
Should individual images be rescaled 0-1 before compensating pre-masking or on unmasked images?:No
104+
Should images be rescaled 0-1 before compensating but after masking to objects?:No
105+
Should histogram matching be performed between the image groups?:No
106+
What compensation class should serve as the template histogram?:1
107+
Should images be rescaled 0-1 after compensating?:No
108+
Should the images be divided by a scalar based on group percentiles:No
109+
What percentile should be used for multiplication:99.0
110+
Should the images have a tophat filter applied before correction?:No
111+
What size radius should be used for the tophat filter?:3
112+
Should the images have a Laplacian of Gaussian filter applied before correction?:No
113+
What size radius should be used for the LoG filter?:1
114+
Should the images have a Difference of Gaussians filter applied before correction?:No
115+
What size sigma should be used for the DoG low sigma?:3
116+
What size radius should be used for the DoG low sigma?:5
117+
118+
DistanceTransform:[module_num:7|svn_version:'Unknown'|variable_revision_number:1|show_window:True|notes:[]|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
119+
Select the input image:DNA
120+
Name the output image:DistanceTransform
121+
Rescale values from 0 to 1?:Yes
122+
123+
EnhancedMeasureTexture:[module_num:8|svn_version:'Unknown'|variable_revision_number:3|show_window:True|notes:[]|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
124+
Hidden:1
125+
Hidden:1
126+
Hidden:1
127+
Select an image to measure:DNA
128+
Select objects to measure:IdentifyPrimaryObjects
129+
Texture scale to measure:3
130+
Angles to measure:Horizontal,Vertical,Diagonal,Anti-diagonal
131+
Measure Gabor features?:Yes
132+
Number of angles to compute for Gabor:4
133+
Measure Tamura features?:Yes
134+
Features to compute:Coarseness,Contrast,Directionality
135+
136+
HistogramEqualization:[module_num:9|svn_version:'Unknown'|variable_revision_number:1|show_window:True|notes:[]|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
137+
Select the input image:DNA
138+
Name the output image:HistogramEqualization
139+
Bins:256
140+
Mask:Leave blank
141+
Local:Yes
142+
Tile Size:100
143+
Clip limit:0.01
144+
Is you image 3D?:No
145+
Do framewise calculation?:No
146+
Tile Size (Z):5
147+
148+
HistogramMatching:[module_num:10|svn_version:'Unknown'|variable_revision_number:1|show_window:True|notes:[]|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
149+
Select the input image:DNA
150+
Name the output image:HistogramMatching
151+
Is you image 3D?:No
152+
Use a frame within image as reference?:No
153+
Image to use as reference :None
154+
Frame number:5
155+
156+
PixelShuffle:[module_num:11|svn_version:'Unknown'|variable_revision_number:1|show_window:True|notes:[]|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
157+
Select the input image:DNA
158+
Name the output image:PixelShuffle

0 commit comments

Comments
 (0)