Merge pull request #269 from Caltech-IPAC/raen/issues/165/spherex-bad-gateway

Catch spherex read errors

Author: troyraen

tutorials/spherex/spherex_cutouts.md

@@ -53,7 +53,9 @@ The following packages must be installed to run this notebook.
 
 ```{code-cell} ipython3
 import concurrent.futures
+import http.client
 import time
+import urllib.error
 
 import astropy.units as u
 import matplotlib.pyplot as plt
@@ -187,6 +189,34 @@ def process_cutout(row, ra, dec, cache):
         row["hdus"] = hdus
 ```
 
+We provide a small convenience wrapper around `process_cutout` that is used in the rest of this notebook.
+It catches transient read errors and simply skips those cutouts, which is sufficient for this demonstration.
+For science use cases, users may instead want to implement their own retry logic or error handling strategy.
+
+```{code-cell} ipython3
+def process_cutout_with_error_handling(row, ra, dec, cache):
+    '''
+    Call `process_cutout` while catching transient read errors.
+
+    Parameters:
+    ===========
+
+    row : astropy.table row
+        Row of a table that will be changed in place by this function. The table
+        is created by the SQL TAP query.
+    ra,dec : coordinates (astropy units)
+        Ra and Dec coordinates (same as used for the TAP query) with attached astropy units
+    cache : bool
+        If set to `True`, the output of cached and the cutout processing will run faster next time.
+        Turn this feature off by setting `cache = False`.
+    '''
+    try:
+        process_cutout(row, ra, dec, cache=cache)
+    except (TimeoutError, urllib.error.HTTPError, http.client.IncompleteRead):
+        # Transient read errors. Skip this cutout.
+        row["hdus"] = None
+```
+
 ## 7. Download the Cutouts
 
 This process can take a while.
@@ -223,8 +253,13 @@ results_table_serial["hdus"] = np.full(len(results_table_serial), None)
 
 t1 = time.time()
 for row in results_table_serial:
-    process_cutout(row, ra, dec, cache=False)
+    process_cutout_with_error_handling(row, ra, dec, cache=False)
 print("Time to create cutouts in serial mode: {:2.2f} minutes.".format((time.time() - t1) / 60))
+
+# Drop rows that failed to download.
+failed_cutouts_serial = results_table_serial[results_table_serial["hdus"] == None]
+print("Failed to get the following cutouts:\n", failed_cutouts_serial["uri"])
+results_table_serial = results_table_serial[results_table_serial["hdus"] != None]
 ```
 
 ### 7.2 Parallel Approach
@@ -257,9 +292,15 @@ results_table_parallel["hdus"] = np.full(len(results_table_parallel), None)
 
 t1 = time.time()
 with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
-    futures = [executor.submit(process_cutout, row, ra, dec, False) for row in results_table_parallel]
+    futures = [executor.submit(process_cutout_with_error_handling, row, ra, dec, False)
+               for row in results_table_parallel]
     concurrent.futures.wait(futures)
 print("Time to create cutouts in parallel mode: {:2.2f} minutes.".format((time.time() - t1) / 60))
+
+# Drop rows that failed to download.
+failed_cutouts_parallel = results_table_parallel[results_table_parallel["hdus"] == None]
+print("Failed to get the following cutouts:\n", failed_cutouts_parallel["uri"])
+results_table_parallel = results_table_parallel[results_table_parallel["hdus"] != None]
 ```
 
 ## 8. Create a summary table HDU with renamed columns
@@ -343,7 +384,7 @@ plt.show()
 
 ## About this notebook
 
-**Updated:** 24 October 2025
+**Updated:** 5 March 2026
 
 **Contact:** [IRSA Helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html) with questions or problems.
 

tutorials/spherex/spherex_intro.md

@@ -60,6 +60,10 @@ The following packages must be installed to run this notebook.
 ## 4. Imports
 
 ```{code-cell} ipython3
+import http.client
+import time
+import urllib.error
+
 import numpy as np
 import matplotlib.pyplot as plt
 
@@ -152,10 +156,26 @@ You can put this URL into a browser to download the file. Or you can work with i
 
 +++
 
-Use Astropy to examine the header of the URL from the previous step.
+First, use Astropy to load the data using the URL from the previous step.
+Transient read errors occur sometimes, so we'll catch those and retry a few times.
+
+```{code-cell} ipython3
+# Max number of times to retry HTTP errors.
+max_retries = 3
+for attempt in range(max_retries):
+    try:
+        # Load the data.
+        hdulist = fits.open(spectral_image_url)
+        break
+    except (TimeoutError, urllib.error.HTTPError, http.client.IncompleteRead):
+        if attempt == max_retries - 1:
+            raise
+        time.sleep(10 * (attempt + 1))
+```
+
+Examine the header.
 
 ```{code-cell} ipython3
-hdulist = fits.open(spectral_image_url)
 hdulist.info()
 ```
 
@@ -454,6 +474,6 @@ print("Dimensions of VALUES array:",  wavelengths.shape)
 **Contact:** [IRSA Helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html) with questions
 or problems.
 
-**Updated:** 24 October 2025
+**Updated:** 5 March 2026
 
 **Runtime:** approximately 30 seconds

tutorials/spherex/spherex_psf.md

@@ -55,8 +55,10 @@ The following packages must be installed to run this notebook.
 ```
 
 ```{code-cell} ipython3
+import http.client
 import re
 import time
+import urllib.error
 
 import astropy.units as u
 import matplotlib.pyplot as plt
@@ -133,18 +135,34 @@ print(spectral_image_url)
 ## 5. Read in a SPHEREx Cutout
 
 Next, we use standard astropy tools to open the fits image and to read the different headers and data.
+Transient read errors occur sometimes, so we'll catch those and retry a few times.
 
 ```{tip}
 As we do below, you can use `hdul.info()` to print the list of FITS layers of the downloaded cutout.
 ```
 
 ```{code-cell} ipython3
-with fits.open(spectral_image_url) as hdul:
-    hdul.info()
-    cutout_header = hdul['IMAGE'].header
-    psf_header = hdul['PSF'].header
-    cutout = hdul['IMAGE'].data
-    psfcube = hdul['PSF'].data
+# Max number of times to retry transient read errors.
+max_retries = 3
+for attempt in range(max_retries):
+    try:
+        # Read the data.
+        with fits.open(spectral_image_url) as hdul:
+            cutout_header = hdul['IMAGE'].header
+            psf_header = hdul['PSF'].header
+            cutout = hdul['IMAGE'].data
+            psfcube = hdul['PSF'].data
+        break
+    except (TimeoutError, urllib.error.HTTPError, http.client.IncompleteRead):
+        if attempt == max_retries - 1:
+            raise
+        time.sleep(10 * (attempt + 1))
+```
+
+Examine the header.
+
+```{code-cell} ipython3
+hdul.info()
 ```
 
 The downloaded SPHEREx image cutout contains 5 FITS layers, which are described in the [SPHEREx Explanatory Supplement](https://irsa.ipac.caltech.edu/data/SPHEREx/docs/SPHEREx_Expsupp_QR.pdf).
@@ -279,11 +297,11 @@ plt.show()
 
 ## 9. Using the SPHEREx PSF in Forward Modeling (e.g., Tractor)
 
-The PSF returned by this notebook is oversampled relative to the native SPHEREx detector pixel grid. 
+The PSF returned by this notebook is oversampled relative to the native SPHEREx detector pixel grid.
 This is intentional: the PSF is evaluated on a fine sub-pixel grid so that it can represent different intra-pixel source positions accurately.
 
-Tools such as Tractor do not expect an oversampled PSF directly. 
-Instead, they require a PSF that is pixel-integrated at the native detector resolution and evaluated at the correct sub-pixel phase of the source. 
+Tools such as Tractor do not expect an oversampled PSF directly.
+Instead, they require a PSF that is pixel-integrated at the native detector resolution and evaluated at the correct sub-pixel phase of the source.
 If you pass the oversampled PSF directly into Tractor without resampling, the effective PSF width and normalization will be incorrect, which can lead to systematic differences relative to the SPHEREx Spectrophotometry Tool.
 
 To use this PSF for forward modeling or fitting, you must:
@@ -299,7 +317,7 @@ To use this PSF for forward modeling or fitting, you must:
 
 ## About this notebook
 
-**Updated:** 24 October 2025
+**Updated:** 5 March 2026
 
 **Contact:** Contact [IRSA Helpdesk](https://irsa.ipac.caltech.edu/docs/help_desk.html) with questions or problems.