Migrate ngmix PSF/shape column names (tracking shapepipe#761)#201
Open
cailmdaley wants to merge 11 commits into
Open
Migrate ngmix PSF/shape column names (tracking shapepipe#761)#201cailmdaley wants to merge 11 commits into
cailmdaley wants to merge 11 commits into
Conversation
Downstream of CosmoStat/shapepipe#761, which makes the ngmix output columns a single source of truth (true original-PSF fit stored separately, ELL split into scalar G1/G2, ESTIMATOR_COMPONENT_OBJECT grammar). Captures the old->new column map and the sp_validation consumer checklist. Blocked on #761 landing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The plan committed 2026-06-20 predated the "drop GAL" decision on
shapepipe#761: it showed every galaxy column gaining a _GAL token and
said nothing about HSM. Rewrite it to the authoritative as-shipped
grammar, grounded in the #761 producer source:
- galaxy is the implicit object (no _GAL token); ellipticity splits into
named scalars NGMIX_G1/G2_{shear}, errors likewise;
- NGMIX_Tpsf -> NGMIX_T_PSF_RECONV (value-safe); ELL_PSFo/T_PSFo ->
G1/G2/T_PSF_ORIG (value change, shapepipe#749);
- HSM E*_{PSF,STAR} -> HSM_G*_{PSF,STAR} (native g, pure rename),
SIGMA_*_HSM -> HSM_T_* (units: now T=2sigma^2), FLAG_*_HSM -> HSM_FLAG_*;
- spread_model removed; sizes route through cs_util.size; square_size
retired. e->g conversion is not needed anywhere (both estimators store g).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8BwwtExg961NknUeu5GCv
Consumer half of shapepipe#761. Every shape-column read in the package
moves onto the one grammar:
- metacal (calibration.py): galaxy ellipticity read as named scalars
NGMIX_G1/G2_{shear} and errors NGMIX_G1/G2_ERR_{shear} (was packed
NGMIX_ELL_*[:, c]); NGMIX_Tpsf -> NGMIX_T_PSF_RECONV. The obsolete
col_2d flag (packed-vs-split ellipticity) is removed from the metacal
ctor and get_variance_ivweights — v2 is always named scalars.
- galaxy.py: NGMIX_ELL_PSFo[:, 0] -> NGMIX_G1_PSF_ORIG scalar; the
spread_model branch/param dropped (shapepipe stopped writing it) —
classification falls back to the size-based path.
- catalog.py: match_spread_class deleted (SPREAD_CLASS gone);
match_subsample takes two scalar keys (g1_key, g2_key) instead of a
packed ell_key; check_invalid drops the now-meaningless comp arg.
- glass_mock.py: HSM E1/E2_PSF_HSM -> HSM_G1/G2_PSF.
- rho_tau.py: sizes are always T=2sigma^2 now, so square_size is False
and the not_square_size list is gone.
- tests: synthetic catalogues + configs rebuilt on the v2 grammar in
lock-step (the migration's internal-consistency check).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8BwwtExg961NknUeu5GCv
…_size
- cat_config.yaml: every HSM block onto the v2 grammar — E*_{PSF,STAR}_HSM
-> HSM_G*_{PSF,STAR}, SIGMA_*_HSM / T_*_HSM -> HSM_T_*, FLAG_*_HSM ->
HSM_FLAG_*. The DES/piff block is untouched (Piff isn't this grammar).
The dead per-dataset square_size flags are dropped: sizes are always
T=2sigma^2 now, so nothing squares (this also fixes a latent factor
error in the SP_v1.3 block, which squared its already-T T_PSF_HSM).
- mask_v1.X.{2..11}.yaml: NGMIX_ELL_PSFo_NOSHEAR_0/_1 ->
NGMIX_G1/G2_PSF_ORIG_NOSHEAR.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8BwwtExg961NknUeu5GCv
…mmar
Non-library consumers onto the same grammar:
- scripts/calibration/{extract_info,params,calibrate_comprehensive_cat}.py:
Tpsf -> T_PSF_RECONV; ELL_PSFo -> G1/G2_PSF_ORIG; T_PSFo -> T_PSF_ORIG;
bare galaxy ELL -> G1/G2 scalars (match_subsample/check_invalid callers
updated for the new signatures); HSM E*_PSF -> HSM_G*_PSF; spread_model
wiring removed; col_2d args dropped.
- scripts/apply_alpha_snr_size_bin.py, scripts/examples/demo_*.py: same
Tpsf/PSFo renames, col_2d dropped.
- cosmo_val/compute_theory_cov.py, cosmo_inference/scripts/masking.py:
Tpsf -> T_PSF_RECONV; square_size hardcoded False.
- papers/catalog/*, papers/harmonic/*: HSM ellipticity renames and the
sigma->T units fix (hand-rolled SIGMA_*_HSM**2 -> read HSM_T_* directly).
Out of scope, left as-is: GALSIM_* columns, scratch/, and the
cosmo_inference notebooks (own migration fiber).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8BwwtExg961NknUeu5GCv
cailmdaley
force-pushed
the
migrate/ngmix-psf-column-names
branch
from
01f9c8f to
c84e404
Compare
Two tightening-pass changes to complete ShapePipe-v2 column adoption in
the package:
- Remove the galsim estimator path as dead code. `shape` is hardcoded to
"ngmix", extract_info.py raises for any other value, and nothing outside
scratch/ instantiates metacal(prefix="GALSIM") or calls
classification_galaxy_galsim. Migrating it would carry an untestable path
whose shared `col_1p = {prefix}_T_PSF_RECONV_1P` read never matched the
galsim producer output (GALSIM_T_PSF_*, not ..._T_PSF_RECONV_*) — already
broken. Removed: metacal._read_data_galsim, the prefix=="GALSIM" dispatch
(now else: raise — unknown prefixes fail loudly), the two galsim
ellipticity sign flips in _shear_response/_selection_response,
galaxy.classification_galaxy_galsim, the sh=="galsim" branch in
catalog.get_snr (now else: raise), and the unused shape_method arg on
get_calibrated_quantities/get_calibrated_m_c. ngmix is the sole estimator;
the `prefix` param stays (names the column family; a future NGMIXm moments
family could reuse it).
- Rename the galaxy failure-flag read NGMIX_MOM_FAIL -> NGMIX_MCAL_TYPES_FAIL
in classification_galaxy_ngmix. The v1 column counted moments-initial-guess
failures (get_guess, gone in v2); the producer reused the slot for a
failed-metacal-types count. sp_validation cuts on == 0 either way (keep
fully-measured objects), so the read migrates cleanly; the underlying
failure mode changed, so this is the first line to check if the post-cut
galaxy count looks off against a regenerated v2 catalogue.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019f8w2twg4b3Ga36dSxNbLW
…figs The v1 failure-flag column no longer exists in the ShapePipe-v2 header; the producer renamed the slot to NGMIX_MCAL_TYPES_FAIL. Propagate the rename into all ten calibration mask configs so the flag column resolves against a v2 catalogue. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019f8w2twg4b3Ga36dSxNbLW
- NGMIX_MOM_FAIL -> NGMIX_MCAL_TYPES_FAIL in params.add_cols_pre_cal (and its int-format set), the two demo column lists, and hist_mag's read lists. - Drop the galsim mentions left in params.py (shape comment) and extract_info.py (the stale "cuts common to ngmix and galsim" comment, whose spread-model line also went with the earlier spread_model removal). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019f8w2twg4b3Ga36dSxNbLW
cfis_analysis.ipynb read E1/E2_PSF_HSM into treecorr for the C_sys PSF test; rename to HSM_G1/G2_PSF (native g, pure rename). A sweep of all tracked notebooks found no other old column tokens. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019f8w2twg4b3Ga36dSxNbLW
Update the authoritative migration map for the tightening pass: - *_PSF_ORIG reframed as a value change that is NOT a code blocker (the rename is correct as-is; a v2 catalogue only enables a look-at-numbers check). The real merge gate is cutover timing — merging makes develop require v2 columns. - NGMIX_MOM_FAIL -> NGMIX_MCAL_TYPES_FAIL added to the ngmix map with a semantics-change note. - galsim path documented as removed dead code (was "left untouched, flagged"). - notebooks added to the consumer-sites list. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019f8w2twg4b3Ga36dSxNbLW
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
shear_psf_leakage #27 removed the square_size parameter from build_cat_to_compute_{rho,tau} and CovTauTh (HSM_T_* already stores the area T=2σ², so the flag was dead weight). sp_validation still threaded square_size into those leakage calls; once #27's container lands, passing it would raise TypeError. Drop the square_size key from both param builders (rho_tau.get_params_rho_tau, cosmo_val/compute_theory_cov.py) and stop passing it to the handlers. Behavior-identical against the current container (square_size defaults to False), required for the post-#27 one. Also correct the migration doc: galsim shapes are not producible by ShapePipe (no shape-measurement runner; make_cat's galsim mode reads an external catalogue nothing generates; production configs pin ngmix) — the decisive reason the dead consumer path was removed rather than migrated; and reframe the shear_psf_leakage coordination now that #27 is a sibling PR landing the same grammar. Suite: 137 passed, 1 skipped (unrelated cmss12.tfm font gap). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GnCCTxKE13cHzPr8BUeKZE
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #218
The sp_validation (consumer) half of the ShapePipe column-grammar unification (CosmoStat/shapepipe#761, landing in #741 / ngmix v2.0). ShapePipe now writes every shape and PSF column under one grammar —
ESTIMATOR_COMPONENT[_ERR][_OBJECT]_SHEAR, uppercase, galaxy implicit (no_GALtoken) — and stores a single sizeT = 2σ². This PR moves every read in sp_validation onto that grammar and drops the machinery the new grammar makes obsolete.The authoritative old→new map is
docs/ngmix_psf_column_migration.md(rewritten here). In brief:NGMIX_ELL_{s}/_ELL_ERR_{s}(2-vec)NGMIX_G1/G2_{s}/_G1/G2_ERR_{s}(named scalars)NGMIX_Tpsf_{s}NGMIX_T_PSF_RECONV_{s}NGMIX_ELL_PSFo_{s}_0/1,NGMIX_T_PSFo_{s}NGMIX_G1/G2_PSF_ORIG_{s},NGMIX_T_PSF_ORIG_{s}E1/2_{PSF,STAR}_HSMHSM_G1/G2_{PSF,STAR}SIGMA_*_HSMHSM_T_*(now the areaT = 2σ²)FLAG_*_HSMHSM_FLAG_*Beyond the renames
T = 2σ². Thesquare_sizemachinery is retired (config flags,not_square_size, both param builders default toFalse) and the hand-rolledSIGMA_*_HSM ** 2squaring is gone from the paper scripts — which also fixes a latent factor error whereSP_v1.3squared an already-Tvalue. Any σ / FWHM / r50 now comes fromcs_util.size, the single source of truth for size math.spread_modelremoved. ShapePipe stopped writingSPREAD_*; thedo_spread_modelbranch andmatch_spread_classare gone, and star/galaxy classification uses the size-based path.GALSIM_*reads were unreachable —shapeis hardcoded tongmixand nothing can produce galsim columns — so the branch is deleted and unknown estimator prefixes now fail loudly. ngmix is the sole estimator.NGMIX_MOM_FAIL→NGMIX_MCAL_TYPES_FAIL. The producer reused the slot; sp_validation cuts on== 0either way, so the read migrates cleanly.col_2dflag is dropped frommetacal/get_variance_ivweights;match_subsampletakes two scalar keys;check_invaliddropscomp.g, so the ellipticity-type step Column-grammar migration: adopt ShapePipe-v2 column names (ngmix_v2.0) #218 flagged isn't needed.One value change to watch
NGMIX_*_PSF_ORIGnow carries a true original-PSF fit (CosmoStat/shapepipe#749), not the reconvolved-kernel alias the oldELL_PSFo/T_PSFocolumns silently held. At the code level this is a straight rename; the look-at-the-numbers check on the α-leakage / size-ratio cuts happens once a v2 catalogue exists — analysis, not code, and it doesn't gate this PR.Merge coordination
Merging makes
developrequire v2 columns and stop reading today's catalogues, so this should land with (or just after) CosmoStat/shapepipe#761 → #741 and the first regenerated catalogue.shear_psf_leakage#27 (separate repo, Sacha) removessquare_sizefrom the ρ/τ internals; this PR setssquare_size=Falseand passes the T-columns, correct independently of that migration.The in-container test suite is green (137 passed; the lone failure is a pre-existing matplotlib TeX-font gap, unrelated to columns).
— Claude on behalf of Cail