ngmix: single source of truth for original vs reconvolved PSF columns

[cailmdaley]

Follows the #741 review discussion of the ngmix PSF columns. The output conflated the original and reconvolved PSF — both *_psf and *_psfo carried the reconvolved fit, so g1_psf (expected ~0 on the metacal image) silently held the original-PSF value, and Tpsf did not mean what its name says. This PR makes each PSF quantity a single source of truth whose name matches its meaning.

Where to focus review

The science lives entirely in ngmix.py — specifically the two average_*_psf weighting functions (how each PSF family is combined across epochs) and the do_ngmix_metacal wiring (the metacal call + the new original-PSF pre-fit and its RNG handling). make_cat.py and the .param files are mechanical serialization of the new column names; the rest is test coverage.

Changes

• Restore an explicit fit to the original PSF and store both the original and reconvolved fits, split by name.
• Rename the ngmix columns to a consistent ESTIMATOR_COMPONENT_OBJECT grammar; update the shipped .param files.
• Drop the dead r50 columns (derivable from T via cs_util.size).
• Route the bare size conversions through cs_util.size.
• Property + unit tests for the new grammar and both PSF families.

Diff at a glance (~2.0k insertions)

• tests ~1.6k — the bulk: grammar/property/unit coverage of both PSF families + failure paths
• python library ~430
• config (.param) ~110
• uv.lock / test-infra ~20

Notes for review

• The original-PSF pre-fit is seeded from a snapshot of the metacal RNG, so it does not advance the RNG metacal consumes — the galaxy/shear results are bit-identical to the pre-PR base; only the new *_PSF_ORIG columns are added. (Rationale in the do_ngmix_metacal comment.)
• Base: ngmix_v2.0 — lands in Ngmix v2.0 (CI mirror of #740) #741 before the v2.0 merge.
• Partially addresses [BUG] ngmix module passes the galaxy shape prior to the PSF fit, which is prior-dominated #749 (the column provenance / naming facet).
• This renames/splits catalogue columns with a real downstream surface in sp_validation — tracked in a separate draft PR there.

— Claude on behalf of Cail

Closes #777

[cailmdaley]

Hello @aguinot @martinkilbinger, this is my first attempt at a fix to all the PSF column issues that cropped up in #741. I decided to be very ambitious and define a common grammar for all column names (not just the PSF columns) to reduce ambiguity; this is quite far reaching since it changes everything that consumes the catalog, and will require a second PR in sp_validation to propagate these changes. However I think it will be worth it and is largely mechanical to propagate. If you think this is too aggressive please let me know and we can consider descoping to only touching the PSF columns. The only real science changes are the new ngmix fit to the original PSF (and the averaging over epochs) in ngmix.py.

Cheers, Cail

[aguinot]

Hello,
I would remove GAL from the ngmix quantities as the objects are not necessarily glaxies and the name let's suppose that a selection has been done and that you have a sample of galaxies only safe to use, while it is not the case. And it also makes the names longer.
You can also keep the spread model up-to-date but, it should not be used to apply a cut as it will create selection effects not taken into account by the response matrix.
I would also use a single size definition everywhere to avoid confusion. If you decide to use the trace T (which I strongly recommend!) only use that and remove columns such as PSF_FWHM (or converrt them to T), it can be derived from T.
Finally, I don't know why I used full capital names in the output catalogue but it might not be a good idea. Feel free to use lower case names.

[sachaguer]

Some notes on the column labelling for the star catalogue. I discussed with @fabianhervaspeters that the HSM shapes are e-types ellipticities, whereas ngmix returns g-type ellipticities. The new labelling shows that, for ngmix, we should ensure the naming is consistent for the HSM quantities and potentially add a function to easily convert between them.

[cailmdaley]

This standardizes the output catalogue onto one column grammar. Every shape estimator the pipeline writes — ngmix, galsim, HSM moments — now names its columns the same way: ESTIMATOR_COMPONENT_OBJECT[_metacaltype] (uppercase), with the galaxy as the implicit default object (no token), ellipticity split into named scalar components (G1/G2 for g-type, E1/E2 for e-type), and a single stored size T = 2σ² from cs_util.size. Any column then parses unambiguously into estimator · quantity · object · metacal shear.

What changed:

• 98e822ad — HSM e/g value fix (below)
• 4b17f7ec — HSM star/PSF grammar + size stored as T
• 85a18ab5 — remove the unused spread_model module
• 4f0756d0 — make_cat grammar for ngmix, galsim & the HSM epoch-sink columns (drop GAL, split packed ellipticity, SIGMA→T, fix a latent n_obj NameError)
• f9a8f050 — track cs_util @ develop for the cs_util.size helpers

Representative renames:

• NGMIX_G1_GAL_NOSHEAR → NGMIX_G1_NOSHEAR (and the rest of the ngmix galaxy block)
• E1_PSF_HSM → HSM_E1_PSF, SIGMA_PSF_HSM → HSM_T_PSF, FLAG_PSF_HSM → HSM_FLAG_PSF (+ STAR variants)
• GALSIM_GAL_ELL_<ext> (packed length-2) → scalar GALSIM_G1_<ext> / GALSIM_G2_<ext>; GALSIM_GAL_SIGMA_<ext> → GALSIM_T_<ext>

One value change (98e822ad). The HSM star/PSF columns are named E* (e-type distortion) but were storing galsim's reduced shear .g1/.g2 (g-type) — name and value disagreed. They now store .e1/.e2, so each column holds the quantity its name says. This is the only change that moves numbers downstream (the ρ/τ star–PSF residual statistics).

— Claude on behalf of Cail

[sachaguer]

I don't understand what Claude says here:

One value change (98e822a). The HSM star/PSF columns are named E* (e-type distortion) but were storing galsim's reduced shear .g1/.g2 (g-type) — name and value disagreed. They now store .e1/.e2, so each column holds the quantity its name says. This is the only change that moves numbers downstream (the ρ/τ star–PSF residual statistics).

I think the HSM columns stored e-types ellipticities that were not converted to g-type. My only point was about having consistent labeling across ngmix shapes and HSM shapes. Can we confirm this?

Edit: I checked the commit. It appears that what we said with Fabian was incorrect and that we saved g-type ellipticities. This is actually fine and what we need for rho/tau-statistics. We could go back to saving g-type and naming the columns appropriately. Would we need to save the e-type as well? Probably not if we have built-in functions to easily make the conversion.

[cailmdaley]

Thanks @sachaguer, this looks great. The argument for recording e for HSM would that HSM natively measures e-type quantities, and since e <-> g is a nonlinear transformation, if your HSM e-type errors are Gaussian, the same wouldn't necessarily be true for your g-type errors. Jensen's inequality would also say their population means would be different. However for PSFs/stars the ellipticity is so small, we have to use g-type for leakage anyway, and as you say we have a lossless conversion function so let's leave it as is.

I think the last thing on this PR is for a shapepipe expert to take a look at the changes to ngmix.py since I added new functionality to average multi-epoch PSF measurements. @martinkilbinger @aguinot @fabianhervaspeters, could one of you please review that when you have a chance? Once this is merged then we should be able to finally merge #741!

[sachaguer]

Agreed on the noise properties. I think this product is mainly used for leakage estimation. If any other use I am not aware of would be simplified by saving both quantities, I don't mind saving both.