Merge pull request #211 from troyraen/raen/issues/191/add-euclid-hats-tutorials-II

Add Euclid HATS Magnitude tutorial 02a9ff9

Author: troyraen

…ld/7bc041dd60f99a861f54b17234d72964.html …ld/09e955ef1ef79aca79fd9ac80bcfb445.htmlbuild/7bc041dd60f99a861f54b17234d72964.html renamed to build/09e955ef1ef79aca79fd9ac80bcfb445.html build/7bc041dd60f99a861f54b17234d72964.html renamed to build/09e955ef1ef79aca79fd9ac80bcfb445.html

@@ -1,5 +1,5 @@
 <div><i>Table length=328</i>
-<table id="table140417987923552" class="table-striped table-bordered table-condensed">
+<table id="table139894057107376" class="table-striped table-bordered table-condensed">
 <thead><tr><th>s_ra</th><th>s_dec</th><th>facility_name</th><th>instrument_name</th><th>dataproduct_subtype</th><th>calib_level</th><th>dataproduct_type</th><th>energy_bandpassname</th><th>energy_emband</th><th>obs_id</th><th>s_resolution</th><th>em_min</th><th>em_max</th><th>em_res_power</th><th>proposal_title</th><th>access_url</th><th>access_format</th><th>access_estsize</th><th>t_exptime</th><th>s_region</th><th>obs_collection</th><th>obs_intent</th><th>algorithm_name</th><th>facility_keywords</th><th>instrument_keywords</th><th>environment_photometric</th><th>proposal_id</th><th>proposal_pi</th><th>proposal_project</th><th>target_name</th><th>target_type</th><th>target_standard</th><th>target_moving</th><th>target_keywords</th><th>obs_release_date</th><th>s_xel1</th><th>s_xel2</th><th>s_pixel_scale</th><th>position_timedependent</th><th>t_min</th><th>t_max</th><th>t_resolution</th><th>t_xel</th><th>obs_publisher_did</th><th>s_fov</th><th>em_xel</th><th>pol_states</th><th>pol_xel</th><th>cloud_access</th><th>o_ucd</th><th>upload_row_id</th><th>dist_to_point</th></tr></thead>
 <thead><tr><th>deg</th><th>deg</th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th>arcsec</th><th>m</th><th>m</th><th></th><th></th><th></th><th></th><th>kbyte</th><th>s</th><th>deg</th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th>arcsec</th><th></th><th>d</th><th>d</th><th>s</th><th></th><th></th><th>deg</th><th></th><th></th><th></th><th></th><th></th><th></th><th></th></tr></thead>
 <thead><tr><th>float64</th><th>float64</th><th>object</th><th>object</th><th>object</th><th>int16</th><th>object</th><th>object</th><th>object</th><th>object</th><th>float64</th><th>float64</th><th>float64</th><th>float64</th><th>object</th><th>object</th><th>object</th><th>int64</th><th>float64</th><th>object</th><th>object</th><th>object</th><th>object</th><th>object</th><th>object</th><th>bool</th><th>object</th><th>object</th><th>object</th><th>object</th><th>object</th><th>bool</th><th>bool</th><th>object</th><th>object</th><th>int64</th><th>int64</th><th>float64</th><th>bool</th><th>float64</th><th>float64</th><th>float64</th><th>int64</th><th>object</th><th>float64</th><th>int64</th><th>object</th><th>int64</th><th>object</th><th>object</th><th>int64</th><th>float64</th></tr></thead>

…-int-68a8143768ea001f8be655ea6858c68c.md …-int-5fe7bfdd92666876d5c89ca197ebf07b.mdbuild/1-euclid-q1-hats-int-68a8143768ea001f8be655ea6858c68c.md renamed to build/1-euclid-q1-hats-int-5fe7bfdd92666876d5c89ca197ebf07b.md build/1-euclid-q1-hats-int-68a8143768ea001f8be655ea6858c68c.md renamed to build/1-euclid-q1-hats-int-5fe7bfdd92666876d5c89ca197ebf07b.md

@@ -1,11 +1,12 @@
 ---
-short_title: "Merged Objects HATS Catalog"
+short_title: Introduction
 jupytext:
   text_representation:
     extension: .md
     format_name: myst
     format_version: 0.13
     jupytext_version: 1.18.1
+  root_level_metadata_filter: -short_title
 kernelspec:
   display_name: Python 3 (ipykernel)
   language: python
@@ -18,16 +19,17 @@ kernelspec:
 
 This tutorial is an introduction to the content and format of the Euclid Q1 Merged Objects HATS Catalog.
 Later tutorials in this series will show how to load quality samples.
+See [Euclid Tutorial Notebooks: Catalogs](../../euclid_access/euclid.md#catalogs) for a list of tutorials in this series.
 
 +++
 
 ## Learning Goals
 
 In this tutorial, we will:
 
-- Learn about the Euclid Merged Objects catalog that IRSA created by combining information from multiple Euclid Quick Release 1 catalogs
+- Learn about the Euclid Merged Objects catalog that IRSA created by combining information from multiple Euclid Quick Release 1 (Q1) catalogs.
 - Find columns of interest.
-- Perform a basic spatial query in each of the Euclid Deep Fields using the Python library PyArrow.
+- Perform a basic query using the Python library PyArrow.
 
 +++
 
@@ -51,12 +53,12 @@ Access is free and no credentials are required.
 
 ## 2. Imports
 
-```{code-cell} python3
+```{code-cell} ipython3
 # # Uncomment the next line to install dependencies if needed.
 # %pip install hpgeom pandas pyarrow
 ```
 
-```{code-cell} python3
+```{code-cell} ipython3
 import hpgeom  # Find HEALPix indexes from RA and Dec
 import pyarrow.compute as pc  # Filter the catalog
 import pyarrow.dataset  # Load the catalog
@@ -70,7 +72,7 @@ First we'll load the Parquet schema (column information) of the Merged Objects c
 The Parquet schema is accessible from a few locations, all of which include the column names and types.
 Here, we load it from the `_common_metadata` file because it also includes the column units and descriptions.
 
-```{code-cell} python3
+```{code-cell} ipython3
 # AWS S3 paths.
 s3_bucket = "nasa-irsa-euclid-q1"
 dataset_prefix = "contributed/q1/merged_objects/hats/euclid_q1_merged_objects-hats/dataset"
@@ -82,7 +84,7 @@ schema_path = f"{dataset_path}/_common_metadata"
 s3 = pyarrow.fs.S3FileSystem(anonymous=True)
 ```
 
-```{code-cell} python3
+```{code-cell} ipython3
 # Load the Parquet schema.
 schema = pyarrow.parquet.read_schema(schema_path, filesystem=s3)
 
@@ -136,7 +138,7 @@ The tables are:
 
 Find all columns from these tables in the Parquet schema:
 
-```{code-cell} python3
+```{code-cell} ipython3
 mer_prefixes = ["mer_", "morph_", "cutouts_"]
 mer_col_counts = {p: len([n for n in schema.names if n.startswith(p)]) for p in mer_prefixes}
 
@@ -193,7 +195,7 @@ The tables are:
 
 Find all columns from these tables in the Parquet schema:
 
-```{code-cell} python3
+```{code-cell} ipython3
 phz_prefixes = ["phz_", "class_", "physparam_", "galaxysed_", "physparamqso_",
                 "starclass_", "starsed_", "physparamnir_"]
 phz_col_counts = {p: len([n for n in schema.names if n.startswith(p)]) for p in phz_prefixes}
@@ -240,7 +242,7 @@ The tables are:
 
 Find all columns from these tables in the Parquet schema:
 
-```{code-cell} python3
+```{code-cell} ipython3
 spe_prefixes = ["z_", "lines_", "models_"]
 spe_col_counts = {p: len([n for n in schema.names if n.startswith(p)]) for p in spe_prefixes}
 
@@ -272,7 +274,7 @@ They are useful for spatial queries, as demonstrated in the Euclid Deep Fields s
 
 The HEALPix, Euclid object ID, and Euclid tile ID columns appear first:
 
-```{code-cell} python3
+```{code-cell} ipython3
 schema.names[:5]
 ```
 
@@ -288,7 +290,7 @@ However, PyArrow automatically makes them available as regular columns when the
 
 The HATS columns appear at the end:
 
-```{code-cell} python3
+```{code-cell} ipython3
 schema.names[-3:]
 ```
 
@@ -297,12 +299,12 @@ schema.names[-3:]
 The subsections above show how to find all columns from a given Euclid table as well as the additional columns.
 Here we show some additional techniques for finding columns.
 
-```{code-cell} python3
+```{code-cell} ipython3
 # Access the data type using the `field` method.
 schema.field("mer_flux_y_2fwhm_aper")
 ```
 
-```{code-cell} python3
+```{code-cell} ipython3
 # The column metadata includes unit and description.
 # Parquet metadata is always stored as bytestrings, which are denoted by a leading 'b'.
 schema.field("mer_flux_y_2fwhm_aper").metadata
@@ -311,7 +313,7 @@ schema.field("mer_flux_y_2fwhm_aper").metadata
 Euclid Q1 offers many flux measurements, both from Euclid detections and from external ground-based surveys.
 They are given in microjanskys, so all flux columns can be found by searching the metadata for this unit.
 
-```{code-cell} python3
+```{code-cell} ipython3
 # Find all flux columns.
 flux_columns = [field.name for field in schema if field.metadata[b"unit"] == b"uJy"]
 
@@ -321,7 +323,7 @@ flux_columns[:4]
 
 Columns associated with external surveys are identified by the inclusion of "ext" in the name.
 
-```{code-cell} python3
+```{code-cell} ipython3
 external_flux_columns = [name for name in flux_columns if "ext" in name]
 print(f"{len(external_flux_columns)} flux columns from external surveys. First four are:")
 external_flux_columns[:4]
@@ -332,14 +334,14 @@ external_flux_columns[:4]
 +++
 
 Euclid Q1 includes data from three Euclid Deep Fields: EDF-N (North), EDF-S (South), EDF-F (Fornax; also in the southern hemisphere).
-There is also a small amount of data from a fourth field: LDN1641 (Lynds' Dark Nebula 1641), which was observed for technical reasons during Euclid's verification phase and mostly ignored here.
+There is also a small amount of data from a fourth field: LDN1641 (Lynds' Dark Nebula 1641), which was observed for technical reasons during Euclid's verification phase.
 The fields are described in [Euclid Collaboration: Aussel et al., 2025](https://arxiv.org/pdf/2503.15302) and can be seen on this [skymap](https://irsa.ipac.caltech.edu/data/download/parquet/euclid/q1/merged_objects/hats/euclid_q1_merged_objects-hats/skymap.png).
 
 The regions are well separated, so we can distinguish them using a simple cone search without having to be too picky about the radius.
 We can load data more efficiently using the HEALPix order 9 pixels that cover each area rather than using RA and Dec values directly.
 These will be used in later tutorials.
 
-```{code-cell} python3
+```{code-cell} ipython3
 # EDF-N (Euclid Deep Field - North)
 ra, dec, radius = 269.733, 66.018, 4  # 20 sq deg
 edfn_k9_pixels = hpgeom.query_circle(hpgeom.order_to_nside(9), ra, dec, radius, inclusive=True)
@@ -360,9 +362,10 @@ ldn_k9_pixels = hpgeom.query_circle(hpgeom.order_to_nside(9), ra, dec, radius, i
 ## 6. Basic Query
 
 To demonstrate a basic query, we'll search for objects with a galaxy photometric redshift estimate of 6.0 (largest possible).
-Other tutorials in this series will show more complex queries and describe the redshifts and other data in more detail.
+Other tutorials in this series will show more complex queries, and describe the redshifts and other data in more detail.
+PyArrow dataset filters are described at [Filtering by Expressions](https://arrow.apache.org/docs/python/compute.html#filtering-by-expressions), and the list of available functions is at [Compute Functions](https://arrow.apache.org/docs/python/api/compute.html).
 
-```{code-cell} python3
+```{code-cell} ipython3
 dataset = pyarrow.dataset.dataset(dataset_path, partitioning="hive", filesystem=s3, schema=schema)
 
 highz_objects = dataset.to_table(
@@ -375,6 +378,6 @@ highz_objects
 
 **Authors:** Troy Raen, Vandana Desai, Andreas Faisst, Shoubaneh Hemmati, Jaladh Singhal, Brigitta Sipőcz, Jessica Krick, the IRSA Data Science Team, and the Euclid NASA Science Center at IPAC (ENSCI).
 
-**Updated:** 2025-12-22
+**Updated:** 2025-12-23
 
 **Contact:** [IRSA Helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html)