Skip to content

Commit 4c28be5

Browse files
troyraentroyraen
authored
Merge pull request #258 from Caltech-IPAC/spherex_sdt_021126
Add SPHEREx Source Discovery Tool
2 parents c681551 + 5ad5332 commit 4c28be5

18 files changed

Lines changed: 2637 additions & 3 deletions

‎deploy_to_fornax_manifest.in‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,19 @@ tutorials/simulated-data/roman_hlss_number_density.md
1818
tutorials/spherex/spherex_cutouts.md
1919
tutorials/spherex/spherex_intro.md
2020
tutorials/spherex/spherex_psf.md
21+
tutorials/spherex/spherex_source_discovery/conda-spherex_sdt.yml
22+
tutorials/spherex/spherex_source_discovery/README.md
23+
tutorials/spherex/spherex_source_discovery/spherex_source_discovery_tool_demo.md
24+
tutorials/spherex/spherex_source_discovery/spherex_source_discovery_tool/__init__.py
25+
tutorials/spherex/spherex_source_discovery/spherex_source_discovery_tool/aperture_photometry.py
26+
tutorials/spherex/spherex_source_discovery/spherex_source_discovery_tool/bokeh_viz.py
27+
tutorials/spherex/spherex_source_discovery/spherex_source_discovery_tool/default_sdt.param
28+
tutorials/spherex/spherex_source_discovery/spherex_source_discovery_tool/default_sdt.sex
29+
tutorials/spherex/spherex_source_discovery/spherex_source_discovery_tool/default.conv
30+
tutorials/spherex/spherex_source_discovery/spherex_source_discovery_tool/default.nnw
31+
tutorials/spherex/spherex_source_discovery/spherex_source_discovery_tool/firefly_viz.py
32+
tutorials/spherex/spherex_source_discovery/spherex_source_discovery_tool/sdt_utils.py
33+
tutorials/spherex/spherex_source_discovery/spherex_source_discovery_tool/source_extraction.py
2134
tutorials/spitzer/siav2_seip.md
2235
tutorials/techniques-and-tools/cloud-access-intro.md
2336
tutorials/techniques-and-tools/Parallelize_Convolution.md

‎ignore_tutorials/ignore_rendering_execution‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,5 @@
22
tutorials/techniques-and-tools/Parallelize_Convolution
33
# Uses about 10G of memory which doesn't leave enough for other notebooks
44
tutorials/wise/wise-allwise-catalog-demo
5+
# Requires source extractor, which requires a conda install
6+
tutorials/spherex/spherex_source_discovery/spherex_source_discovery_tool_demo.md

‎notebook_metadata.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,9 @@
7676
- title: SIR 1D Spectra
7777
file: tutorials/euclid/3_Euclid_intro_1D_spectra.md
7878
description: Load a galaxy spectrum and plot it. Understand the wavelength, flux, and mask values.
79+
- title: Source Discovery Tool
80+
file: tutorials/spherex/spherex_source_discovery/spherex_source_discovery_tool_demo.md
81+
description: Discover, extract, and visualize sources from SPHEREx Spectral Images.
7982
- title: Spectral Image Cutouts
8083
file: tutorials/spherex/spherex_cutouts.md
8184
description: Generate and work with spatial and spectral cutouts.

‎toc.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@ project:
1212
file: tutorials/spherex/spherex_cutouts.md
1313
- title: PSF Models
1414
file: tutorials/spherex/spherex_psf.md
15+
- title: Source Discovery Tool
16+
file: tutorials/spherex/spherex_source_discovery/spherex_source_discovery_tool_demo.md
1517
- title: Euclid
1618
file: tutorials/euclid/euclid.md
1719
children:

‎tutorials/spherex/spherex.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
1-
# Spherex Tutorial Notebooks
1+
# SPHEREx Tutorial Notebooks
22

3-
[SPHEREx](https://irsa.ipac.caltech.edu/Missions/spherex.html) (Spectro-Photometer for the History of the Universe, Epoch of Reionization, and Ices Explorer) is a NASA space mission designed to perform the first all-sky spectral survey in the near-infrared.
4-
SPHEREx observes the sky from roughly 0.75–5.0 µm using a single instrument that provides low-resolution spectroscopy (R ≈ 40–150) in hundreds of spectral channels for every point on the sky.
3+
[SPHEREx](https://irsa.ipac.caltech.edu/Missions/spherex.html) (Spectro-Photometer for the History of the Universe, Epoch of Reionization, and Ices Explorer) is a NASA space mission designed to perform the first all-sky spectral survey in the near-infrared.
4+
SPHEREx observes the sky from roughly 0.75–5.0 µm using a single instrument that provides low-resolution spectroscopy (R ≈ 40–150) in hundreds of spectral channels for every point on the sky.
55
Its science goals span cosmology, galaxy evolution, and the interstellar medium, enabling measurements of large-scale structure, the cosmic history of star formation, and the distribution of key molecules and ices in the Milky Way and nearby galaxies.
66

77
SPHEREx data releases include weekly [Quick Release spectral image products](https://caltech-ipac.github.io/spherex-archive-documentation/spherex-data-products/) (multi-extension FITS files containing calibrated near-infrared surface brightness, variance, flags, modeled backgrounds, PSFs, and wavelength WCS) along with ancillary calibration and metadata files such as gain matrices, dark current maps, solid angle pixel maps, and detailed spectral WCS products for each detector.
@@ -12,3 +12,4 @@ SPHEREx data releases include weekly [Quick Release spectral image products](htt
1212

1313
- [PSF Models](spherex_psf.md) - Understand how SPHEREx point spread function (PSF) information is organized and accessed.
1414

15+
- [Source Discovery Tool](spherex_source_discovery/spherex_source_discovery_tool_demo.md) - Discover, extract, and visualize sources from SPHEREx Spectral Images.

‎tutorials/spherex/spherex_source_discovery/README.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# SPHEREx Source Discovery Tool
2+
3+
## Overview
4+
5+
The SPHEREx Source Discovery Tool is the Python package `spherex_source_discovery_tool` (included in this directory), which is used to discover and extract sources from SPHEREx Spectral Images and visualize their spectra.
6+
The notebook [](spherex_source_discovery_tool_demo.md) demonstrates how to use it.
7+
8+
### Directory Contents
9+
10+
- `spherex_source_discovery_tool_demo.md`: Demo notebook.
11+
- `conda-spherex_sdt.yml`: Conda environment definition file.
12+
- `spherex_source_discovery_tool/`: Python package, including functions and configuration files to extract and visualize sources.
13+
- Python modules
14+
- `aperture_photometry.py`: Functions adapted from Zafar Rustamkulov to perform aperture photometry.
15+
- `bokeh_viz.py`: Functions for dynamic `bokeh` visualizations.
16+
- `firefly_viz.py`: Functions for `Firefly` visualizations.
17+
- `source_extraction.py`: Functions to work with SExtractor.
18+
- `sdt_utils.py`: Miscellaneous utility functions for working with IRSA tables and SPHEREx image headers.
19+
- SExtractor files
20+
- `default.conv`: Convolution mask used as detection filter.
21+
- `default.nnw`: Table of neural-network weights for star/galaxy separation.
22+
- `default_sdt.sex`: SExtractor default config file. Specifies catalog, extraction, photometry, etc. options.
23+
- `default_sdt.param`: SExtractor default parameters file. Specifies columns to include in output SExtractor catalog.
24+
25+
## Setup
26+
27+
### On local machine
28+
29+
All necessary packages and tools (SExtractor) are listed in the `conda-spherex_sdt.yml` file.
30+
31+
To create a conda environment with the dependencies on your local machine use:
32+
```
33+
conda env create --file conda-spherex_sdt.yml
34+
```
35+
36+
Then, make the environment available in the list of kernels for your JupyterLab:
37+
```
38+
conda activate spherex_sdt
39+
python -m ipykernel install --user --name=spherex_sdt
40+
```
41+
42+
To use the environment in your Jupyter notebooks, either start JupyterLab in that environment by typing
43+
44+
```
45+
conda activate spherex_sdt
46+
jupyter-lab
47+
```
48+
49+
or select the environment `spherex_sdt` in your Jupyter Notebook using the dropdown on the upper left.
50+
51+
### On Fornax
52+
53+
Installing the conda environment on the [NASA Fornax Science Console](https://science.nasa.gov/astrophysics/programs/physics-of-the-cosmos/community/the-fornax-initiative/) needs slightly different steps. These can be reviewed in the documentation [create a new environment](https://docs.fornax.sciencecloud.nasa.gov/compute-environments/#create-new-env).
54+
55+
In order to install this specific conda environment on Fornax, the file name of the `yml` file specifically needs to be in the following format `conda-*.yml`. The `yml` file distributed here is already in that format (`conda-spherex_sdt.yml`). Once this is set, open a new terminal (click on the large "+" button right under the "File" menu tab) and type to following command _inside_ the same directory where the `yml` file is located:
56+
```
57+
setup-conda-env --user
58+
```
59+
60+
Note that we use the `--user` option here, which will keep the environment available for subsequent Fornax sessions.
61+
62+
To use the environment in your Jupyter notebooks on Fornax, directly select the environment `spherex_sdt` in your Jupyter Notebook using the dropdown on the upper left.

‎tutorials/spherex/spherex_source_discovery/conda-spherex_sdt.yml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
name: spherex_sdt
2+
channels:
3+
- conda-forge
4+
dependencies:
5+
- astromatic-source-extractor
6+
- astropy
7+
- astroquery
8+
- bokeh
9+
- ipykernel
10+
- ipywidgets
11+
- matplotlib
12+
- numpy
13+
- pandas
14+
- photutils
15+
- pip
16+
- regions
17+
- reproject
18+
- scipy
19+
- sep
20+
- tqdm
21+
- pip:
22+
- firefly-client

‎tutorials/spherex/spherex_source_discovery/spherex_source_discovery_tool/__init__.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
from .aperture_photometry import *
2+
from .bokeh_viz import *
3+
from .firefly_viz import *
4+
from .sdt_utils import *
5+
from .source_extraction import *

0 commit comments

Comments
 (0)