Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
b6c84c4
ngmix: restore original-PSF fit; split the two PSF families by name (…
cailmdaley Jun 19, 2026
7e7887b
make_cat: rename ngmix columns to the ESTIMATOR_COMPONENT_OBJECT gram…
cailmdaley Jun 19, 2026
c3fbebe
tests: cover both PSF families and the 3-tuple do_ngmix_metacal (#749)
cailmdaley Jun 19, 2026
b7ababa
make_cat: update shipped param files to the new ngmix column grammar …
cailmdaley Jun 19, 2026
9b3025d
make_cat: add _save_ngmix_data tests; soften the grammar docstring (#…
cailmdaley Jun 19, 2026
9ab43af
ngmix: fit a copy of the original PSF so gal_obs.psf is never mutated…
cailmdaley Jun 19, 2026
b690d3f
ngmix: drop dead r50 columns (derivable from T)
cailmdaley Jun 19, 2026
85db8c1
tests: hypothesis property tests for the PSF column refactor
cailmdaley Jun 19, 2026
dfac162
deps: bump cs_util 0.2 -> 0.2.1 so cs_util.size is in the image
cailmdaley Jun 19, 2026
413c5dd
refactor: route size conversions through cs_util.size
cailmdaley Jun 19, 2026
e2f37be
test: make_data gains psf_shear for an elliptical PSF
cailmdaley Jun 20, 2026
31d2080
review: address PR #761 code review
cailmdaley Jun 20, 2026
8ae3203
refactor(ngmix): build the per-family PSF res-keys from one loop
cailmdaley Jun 20, 2026
2087874
refactor(make_cat): family-loop the PSF column write block
cailmdaley Jun 20, 2026
ade9cf3
docs(CLAUDE): PYTHONPATH-shadow recipe for testing a library build in…
cailmdaley Jun 30, 2026
98e822a
fix(hsm): store true e-type distortion in E*_HSM columns, not g-type …
cailmdaley Jun 30, 2026
4b17f7e
refactor(hsm): one column grammar for HSM star/PSF moments, size stor…
cailmdaley Jun 30, 2026
85a18ab
refactor: remove the spread_model pipeline module (unused, never cut on)
cailmdaley Jun 30, 2026
4f0756d
refactor(make_cat): one column grammar for ngmix, galsim & HSM-sink c…
cailmdaley Jun 30, 2026
f9a8f05
build: track cs_util @ develop (git ref) for the size/shape helpers
cailmdaley Jun 30, 2026
d5b3e92
Go back to g-type ellipticities
sachaguer Jul 1, 2026
7d527d3
Rename the HSM columns with G1/2 instead of E1/2
sachaguer Jul 1, 2026
0df4f2c
(fix) Remaining E1/2 HSM column instead of G1/2
sachaguer Jul 1, 2026
726207b
(fix) Change the HSM label in the associated pytest
sachaguer Jul 1, 2026
bcf4def
(fix) Non-passing test because test was based on e-type ellipticity i…
sachaguer Jul 1, 2026
c5b428b
(fix) Fix the magnitude comparison test between e- and g-type ellipti…
sachaguer Jul 1, 2026
66b338d
ngmix: give the PSF observation a finite weight map (#749) (#779)
cailmdaley Jul 1, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,30 @@ way to get all of that is the container.
sandbox with a host clone of the repo bind-mounted in and `pip install -e`
pointed at it, so edits on the host are live inside the container.

**Shadow a library build with `PYTHONPATH` (no image rebuild).** When you need
the container to run a *different* build of a pure-Python library than the one
baked into its venv — a feature worktree, an unreleased branch, this repo's own
`src/` against a stale image — prepend the host checkout to `PYTHONPATH`. Python
resolves the prepended path first, so the on-disk version shadows
`/app/.venv/...` without touching the image. This is the local-testing
counterpart of a git-ref dependency (e.g. `cs_util @ develop` in
`pyproject.toml`): the dep change makes CI build the right version; the shadow
lets you test that version *now*, before any rebuild. The recipe:

```bash
apptainer exec --bind /n17data,/automnt <image.sif> bash -c \
"cd <repo-worktree> && \
PYTHONPATH=/path/to/libfoo-checkout:<repo-worktree>/src \
python -m pytest <targets> -o addopts='' -q"
```

Notes: the checkout path is the **parent** of the importable package dir (the
dir containing `foo/`, not `foo/` itself); list several `:`-separated to stack
shadows; `-o addopts=''` clears `pyproject.toml`'s pytest defaults when a plugin
they reference (e.g. `pytest-cov`) isn't in the image. Use this for a quick
verify; land the real fix as the `pyproject.toml` / `uv.lock` dep change so CI
and the next image agree.

**Testing container changes: build remotely, pull locally.** Don't
`apptainer build` images on a cluster — quotas are tight and the build is slow.
The loop for any change to `Dockerfile` / `pyproject.toml` / `uv.lock` is: edit
Expand Down
11 changes: 5 additions & 6 deletions docs/source/pipeline_tutorial.md
Original file line number Diff line number Diff line change
Expand Up @@ -289,16 +289,15 @@ job_sp TILE_ID -j 16
is the selection of galaxies as extended objects compared to the PSF.
First, the PSF model is interpolated to galaxy positions, according to the PSF model
with `psfex_interp` or `mccd_interp`. Next, postage stamps around galaxies
of the weights maps are created via `vignetmaker`. Then, the spread model
is computed by the `spread_model` module. Finally, postage stamps
of the weights maps are created via `vignetmaker`. Finally, postage stamps
around galaxies of single-exposure data is extracted with another call
to `vignetmaker`.

The output directory is
- `run_sp_MiViSmVi` if the PSF model is `mccd`;
- `run_sp_tile_PsViSmVi` for the `PSFEx` PSF model.
- `run_sp_MiViVi` if the PSF model is `mccd`;
- `run_sp_tile_PsViVi` for the `PSFEx` PSF model.

This corresponds to the MCCD/PSFex interpolation (`Mi`/`Pi`), `vignetmaker` (`Vi`), `spread_model` (`Sm`), and the
This corresponds to the MCCD/PSFex interpolation (`Mi`/`Pi`), `vignetmaker` (`Vi`), and the
second call to `vignetmaker` (`Vi`).


Expand All @@ -325,7 +324,7 @@ job_sp TILE_ID -j 64
This task first merges the `NJOB` parallel `ngmix` output files from the previous step into
one output file. Then, previously obtained information are pasted into a _final_ shape catalogue via `make_cat`.
Included are galaxy detection and basic measurement parameters, the PSF model at
galaxy positions, the spread-model classification, and the shape measurement.
galaxy positions, and the shape measurement.

Two output directories are created.
The first one is `run_sp_Ms` for the `merge_sep` run.
Expand Down
13 changes: 3 additions & 10 deletions example/cfis/config_make_cat_mccd.ini
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ LOG_NAME = log_sp
RUN_LOG_NAME = log_run_sp

# Input directory, containing input files, single string or list of names with length matching FILE_PATTERN
INPUT_DIR = last:sextractor_runner_run_1, last:spread_model_runner, last:mccd_interp_runner, last:merge_sep_cats_runner
INPUT_DIR = last:sextractor_runner_run_1, last:mccd_interp_runner, last:merge_sep_cats_runner

# Output directory
OUTPUT_DIR = ./output
Expand All @@ -56,22 +56,15 @@ TIMEOUT = 96:00:00

# Input file pattern(s), list of strings with length matching number of expected input file types
# Cannot contain wild cards
FILE_PATTERN = sexcat, sexcat_sm, galaxy_psf, ngmix
#, galsim
FILE_PATTERN = sexcat, galaxy_psf, ngmix

# FILE_EXT (optional) list of string extensions to identify input files
FILE_EXT = .fits, .fits, .sqlite, .fits
#, .fits
FILE_EXT = .fits, .sqlite, .fits

# Numbering convention, string that exemplifies a numbering pattern.
# Matches input single exposures (with 'p' removed)
# Needs to be given in this section, will be updated in module
# sections below
NUMBERING_SCHEME = -000-000

SM_DO_CLASSIFICATION = True
SM_STAR_THRESH = 0.003
SM_GAL_THRESH = 0.01

SHAPE_MEASUREMENT_TYPE = ngmix
#, galsim
10 changes: 3 additions & 7 deletions example/cfis/config_make_cat_psfex.ini
Original file line number Diff line number Diff line change
Expand Up @@ -55,23 +55,19 @@ TIMEOUT = 96:00:00
[MAKE_CAT_RUNNER]

# Input directory, containing input files, single string or list of names with length matching FILE_PATTERN
INPUT_DIR = run_sp_tile_Sx:sextractor_runner, last:spread_model_runner, last:psfex_interp_runner, last:merge_sep_cats_runner
INPUT_DIR = run_sp_tile_Sx:sextractor_runner, last:psfex_interp_runner, last:merge_sep_cats_runner

# Input file pattern(s), list of strings with length matching number of expected input file types
# Cannot contain wild cards
FILE_PATTERN = sexcat, sexcat_sm, galaxy_psf, ngmix
FILE_PATTERN = sexcat, galaxy_psf, ngmix

# FILE_EXT (optional) list of string extensions to identify input files
FILE_EXT = .fits, .fits, .sqlite, .fits
FILE_EXT = .fits, .sqlite, .fits

# Numbering convention, string that exemplifies a numbering pattern.
# Matches input single exposures (with 'p' removed)
# Needs to be given in this section, will be updated in module
# sections below
NUMBERING_SCHEME = -000-000

SM_DO_CLASSIFICATION = True
SM_STAR_THRESH = 0.003
SM_GAL_THRESH = 0.01

SHAPE_MEASUREMENT_TYPE = ngmix
2 changes: 0 additions & 2 deletions example/cfis/config_make_cat_psfex_nosm.ini
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,4 @@ FILE_EXT = .fits, .sqlite, .fits
# sections below
NUMBERING_SCHEME = -000-000

SM_DO_CLASSIFICATION = False

SHAPE_MEASUREMENT_TYPE = ngmix
56 changes: 30 additions & 26 deletions example/cfis/final_cat.param
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,9 @@ FLAGS
IMAFLAGS_ISO
NGMIX_MCAL_FLAGS

# PSF ellipticity
NGMIX_ELL_PSFo_NOSHEAR

# spread class
#SPREAD_CLASS

# spread model flag and error
#SPREAD_MODEL
#SPREADERR_MODEL
# PSF ellipticity (original image PSF)
NGMIX_G1_PSF_ORIG_NOSHEAR
NGMIX_G2_PSF_ORIG_NOSHEAR

# Number of epochs (exposures)
N_EPOCH
Expand All @@ -29,16 +23,26 @@ NGMIX_N_EPOCH
## Ngmix: model fitting

# galaxy ellipticity
NGMIX_ELL_1M
NGMIX_ELL_1P
NGMIX_ELL_2M
NGMIX_ELL_2P
NGMIX_ELL_NOSHEAR
#NGMIX_ELL_ERR_1M
#NGMIX_ELL_ERR_1P
#NGMIX_ELL_ERR_2M
#NGMIX_ELL_ERR_2P
NGMIX_ELL_ERR_NOSHEAR
NGMIX_G1_1M
NGMIX_G2_1M
NGMIX_G1_1P
NGMIX_G2_1P
NGMIX_G1_2M
NGMIX_G2_2M
NGMIX_G1_2P
NGMIX_G2_2P
NGMIX_G1_NOSHEAR
NGMIX_G2_NOSHEAR
#NGMIX_G1_ERR_1M
#NGMIX_G2_ERR_1M
#NGMIX_G1_ERR_1P
#NGMIX_G2_ERR_1P
#NGMIX_G1_ERR_2M
#NGMIX_G2_ERR_2M
#NGMIX_G1_ERR_2P
#NGMIX_G2_ERR_2P
NGMIX_G1_ERR_NOSHEAR
NGMIX_G2_ERR_NOSHEAR

# flags
NGMIX_FLAGS_1M
Expand All @@ -58,11 +62,11 @@ NGMIX_T_ERR_1P
NGMIX_T_ERR_2M
NGMIX_T_ERR_2P
NGMIX_T_ERR_NOSHEAR
NGMIX_Tpsf_1M
NGMIX_Tpsf_1P
NGMIX_Tpsf_2M
NGMIX_Tpsf_2P
NGMIX_Tpsf_NOSHEAR
NGMIX_T_PSF_RECONV_1M
NGMIX_T_PSF_RECONV_1P
NGMIX_T_PSF_RECONV_2M
NGMIX_T_PSF_RECONV_2P
NGMIX_T_PSF_RECONV_NOSHEAR

# flux and error
NGMIX_FLUX_1M
Expand Down Expand Up @@ -94,10 +98,10 @@ FWHM_IMAGE
FWHM_WORLD

# PSF size measured on original image
NGMIX_T_PSFo_NOSHEAR
NGMIX_T_PSF_ORIG_NOSHEAR

# PSF size measured on reconvolved image
# NGMIX_Tpsf_NOSHEAR
# NGMIX_T_PSF_RECONV_NOSHEAR

# ngmix moment failure flag
NGMIX_MOM_FAIL
56 changes: 30 additions & 26 deletions example/unions_800/cat_matched.param
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,9 @@ FLAGS
IMAFLAGS_ISO
NGMIX_MCAL_FLAGS

# PSF ellipticity
NGMIX_ELL_PSFo_NOSHEAR

# spread class
SPREAD_CLASS

# spread model flag and error
SPREAD_MODEL
SPREADERR_MODEL
# PSF ellipticity (original image PSF)
NGMIX_G1_PSF_ORIG_NOSHEAR
NGMIX_G2_PSF_ORIG_NOSHEAR

# Number of epochs (exposures)
N_EPOCH
Expand All @@ -29,16 +23,26 @@ NGMIX_N_EPOCH
## Ngmix: model fitting

# galaxy ellipticity
NGMIX_ELL_1M
NGMIX_ELL_1P
NGMIX_ELL_2M
NGMIX_ELL_2P
NGMIX_ELL_NOSHEAR
#NGMIX_ELL_ERR_1M
#NGMIX_ELL_ERR_1P
#NGMIX_ELL_ERR_2M
#NGMIX_ELL_ERR_2P
NGMIX_ELL_ERR_NOSHEAR
NGMIX_G1_1M
NGMIX_G2_1M
NGMIX_G1_1P
NGMIX_G2_1P
NGMIX_G1_2M
NGMIX_G2_2M
NGMIX_G1_2P
NGMIX_G2_2P
NGMIX_G1_NOSHEAR
NGMIX_G2_NOSHEAR
#NGMIX_G1_ERR_1M
#NGMIX_G2_ERR_1M
#NGMIX_G1_ERR_1P
#NGMIX_G2_ERR_1P
#NGMIX_G1_ERR_2M
#NGMIX_G2_ERR_2M
#NGMIX_G1_ERR_2P
#NGMIX_G2_ERR_2P
NGMIX_G1_ERR_NOSHEAR
NGMIX_G2_ERR_NOSHEAR

# flags
NGMIX_FLAGS_1M
Expand All @@ -58,11 +62,11 @@ NGMIX_T_ERR_1P
NGMIX_T_ERR_2M
NGMIX_T_ERR_2P
NGMIX_T_ERR_NOSHEAR
NGMIX_Tpsf_1M
NGMIX_Tpsf_1P
NGMIX_Tpsf_2M
NGMIX_Tpsf_2P
NGMIX_Tpsf_NOSHEAR
NGMIX_T_PSF_RECONV_1M
NGMIX_T_PSF_RECONV_1P
NGMIX_T_PSF_RECONV_2M
NGMIX_T_PSF_RECONV_2P
NGMIX_T_PSF_RECONV_NOSHEAR

# flux and error
NGMIX_FLUX_1M
Expand All @@ -87,10 +91,10 @@ MAG_AUTO
SNR_WIN

# PSF size measured on original image
NGMIX_T_PSFo_NOSHEAR
NGMIX_T_PSF_ORIG_NOSHEAR

# PSF size measured on reconvolved image
# NGMIX_Tpsf_NOSHEAR
# NGMIX_T_PSF_RECONV_NOSHEAR

# ngmix moment failure flag
NGMIX_MOM_FAIL
Expand Down
7 changes: 6 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ dependencies = [
"astropy>=7.0", # major 6 → 7
"astroquery",
"canfar",
"cs_util>=0.1.9",
"cs_util>=0.2.1",
"galsim>=2.8",
"h5py",
"joblib>=1.4",
Expand Down Expand Up @@ -90,6 +90,11 @@ environments = ["sys_platform == 'linux'"]

[tool.uv.sources]
ngmix = { git = "https://github.com/esheldon/ngmix", tag = "v2.4.0" }
# Track cs_util's develop branch (a git ref, not a pinned release) so new
# cs_util.size / cs_util.shape helpers are available without shipping a release
# and rebuilding the image each time. shapepipe's column grammar sources size
# (T = 2*sigma^2) and ellipticity conversions from cs_util.
cs_util = { git = "https://github.com/CosmoStat/cs_util", branch = "develop" }

[build-system]
# Declaring a build backend makes shapepipe an installed package (not a uv
Expand Down
20 changes: 2 additions & 18 deletions src/shapepipe/modules/make_cat_package/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
:Parent modules:
- ``sextractor_runner``
- ``spread_model_runner``
- ``psfex_interp_runner`` or ``mccd_interp_runner``
- ``ngmix_runner``
Expand All @@ -21,27 +20,12 @@
This module creates a *final* catalogue combining the output of various
previous module runs. This gathers all relevant information on the measured
galaxies for weak-lensing post-processing. This includes galaxy detection and
basic measurement parameters, the PSF model at galaxy positions, the
spread-model classification, and the shape measurement.
basic measurement parameters, the PSF model at galaxy positions, and the
shape measurement.
Module-specific config file entries
===================================
SM_DO_CLASSIFICATION : bool, optional
Adds spread-model star/galaxy classification flag as the column
``SPREAD_CLASS`` to the output if ``True``
SM_STAR_THRESH : float, optional
Threshold :math:`s_{\rm star, thresh}` for star selection; object is
classified as a star if
:math:`|x s + 2 \sigma_s | < s_{\textrm{star, thresh}}`
where :math:`s` is the spread model and :math:`\sigma_s` is the spread
model error; default value is ``0.003``
SM_GAL_THRESH : float, optional
Threshold :math:`s_{\rm gal, thresh}` for galaxy selection; object is
classified as a galaxy if
:math:`s + 2 \sigma_s > s_{\textrm{gal, thresh}}` where :math:`s` is the
spread model and :math:`\sigma_s` is the spread model error; default value
is ``0.01``
SHAPE_MEASUREMENT_TYPE : list
Shape measurement method, valid options are ``ngmix`` and/or ``galsim``
SAVE_PSF_DATA : bool, optional
Expand Down
Loading