MAST: Read product function for MastMissions by snbianco · Pull Request #3593 · astropy/astroquery

snbianco · 2026-05-18T19:54:26Z

A function in the MastMissions class that can read products into memory.

For FITS files, we use astropy.io.fits.open. The product files are downloaded and cached by Astropy, and then read into a HDUList object.

For ASDF files, we use fsspec and asdf. Our downloads for Roman data are redirected to S3, so these files are not downloaded and are streamed directly into asdf.AsdfFile objects.

codecov · 2026-05-18T19:59:07Z

Codecov Report

❌ Patch coverage is 92.00000% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.94%. Comparing base (e93dabb) to head (6d145c7).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
astroquery/mast/missions.py	92.00%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3593      +/-   ##
==========================================
+ Coverage   73.87%   73.94%   +0.06%     
==========================================
  Files         228      228              
  Lines       21485    21572      +87     
==========================================
+ Hits        15873    15952      +79     
- Misses       5612     5620       +8

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

scfleming · 2026-05-20T19:57:19Z

Until we have a more mature version ready, can we please ensure documentation / tutorials call out that for FITS files read_product() still downloads the file locally, so people working on Roman work who may want to use this to download HST or JWST FITS files aren't surprised if read_product() is using up credits writing to local cloud storage on them?

I presume this can be avoided once there's a "get_s3_uris()" endpoint in MissionMAST for all the supported missions beyond Roman?

bsipocz

code changes look good, but I have problem with the proposed changes in pyproject.toml, IMO we should avoid workaround for [all] and should not include optional dependencies to be installed for all the test jobs.

bsipocz · 2026-07-11T02:07:54Z

   "sphinx-astropy>=1.5",
   "scipy",
   "fsspec[http]",
+   "asdf"


where is it used in the docs?

I think I thought I had to include it for the code blocks. Removed!

bsipocz · 2026-07-11T02:09:10Z

-   "fsspec[http]",
+   "fsspec[http,s3]",
   "moto[s3]",
-   "s3fs",
+   "asdf"


I don't think that optional dependencies should be installed for all test runs. The fact that we have fsspec and s3fs listed here is a problem of its own and should be cleaned up. So, I would not think we should add asdf.

snbianco · 2026-10-03T00:47:29Z

Could we get another round of review on this PR? After the weekend, of course. We're hoping to get this particular feature into a dev release as soon as we can since it will be important for accessing Roman data.

bsipocz

Here are some comments, I don't see any big issues but am still waiting on the tests to finish

bsipocz · 2026-08-13T22:56:10Z

+    if asdf is not None:
+        mock_asdf_file = MagicMock(spec=asdf.AsdfFile)
+    else:
+        mock_asdf_file = MagicMock()


I'm not sure what this suppose to be, what is the scenario when you end up in the else part? (if there is no asdf then we should not get to that line)

bsipocz · 2026-08-13T23:02:05Z

+    assert isinstance(obj, fits.HDUList)
+
+
+def test_missions_read_product_asdf(mocker, mock_asdf_open):


do we want to instead decorate these with a skipif like we do for other optional dependencies?

bsipocz · 2026-11-03T05:47:35Z


        return manifest

+    def read_product(self, uri, *, mission=None, **kwargs):


in other modules we tend to have the naming of get_... rather than read_..., but I can't claim we are overly consistent, and that what we have is the way should be. But, eventually we may want to be more unified. And we already have get_product etc in euclid that downloads data.

We chatted about this: renaming and adding a deprecation warning for the rename from read_product sounds good to me.

bsipocz · 2026-11-03T05:54:48Z

+                **file_kwargs,
+            )
+except ImportError:
+    fsspec = None


Minor nitpicking about consistency, we tend to use the HAS_{packagename} variables for cases like this.

bsipocz

Thanks!

snbianco added the mast label May 19, 2026

snbianco force-pushed the read-product-mm branch from f5863dc to 57f3694 Compare May 19, 2026 18:52

snbianco force-pushed the read-product-mm branch 2 times, most recently from 617201a to cdda4a2 Compare May 27, 2026 17:05

snbianco marked this pull request as ready for review May 27, 2026 20:55

snbianco force-pushed the read-product-mm branch from 7808985 to e1ea60e Compare June 12, 2026 15:39

snbianco force-pushed the read-product-mm branch from e1ea60e to a5214b2 Compare July 10, 2026 19:03

bsipocz requested changes Jul 11, 2026

View reviewed changes

snbianco force-pushed the read-product-mm branch from 9eae055 to 2a83677 Compare August 12, 2026 20:02

snbianco force-pushed the read-product-mm branch from 2a83677 to 4e9dd36 Compare August 26, 2026 00:54

snbianco force-pushed the read-product-mm branch from 4e9dd36 to bcd4eb8 Compare September 2, 2026 17:08

snbianco force-pushed the read-product-mm branch 2 times, most recently from f071d04 to d620a7a Compare September 24, 2026 15:28

snbianco force-pushed the read-product-mm branch from d620a7a to f051f27 Compare September 30, 2026 18:52

snbianco force-pushed the read-product-mm branch from 3a4c60f to badbbcc Compare October 27, 2026 17:21

snbianco added 12 commits November 3, 2026 13:21

MastMissions.read_product

e2c5f6a

Docs

150c62a

Adding depencies, changelog

5dc4d76

Authentication fix

fe77ac2

error message phrasing, add guard to token,

bbc693a

Update docs, fix oldestdeps test

d9aa678

Fix docs build

55de5ae

Style fix

5698744

Close objects in remote test

c18cf7d

Walking back pyproject changes

b8ce502

Removing unneeded dependencies

c28a6b3

Style on test dependencies

b853608

snbianco added 3 commits November 3, 2026 13:21

Refresh S3 URL if expired

d666cda

Test fix

db77d29

Better coverage on MastMissions tests

437b9e2

snbianco force-pushed the read-product-mm branch from badbbcc to 437b9e2 Compare November 3, 2026 05:21

bsipocz reviewed Nov 3, 2026

View reviewed changes

Renaming function, adding deprecated function, pytest markers

6d145c7

bsipocz approved these changes Nov 3, 2026

View reviewed changes

bsipocz added this to the 0.4.12 milestone Nov 3, 2026

bsipocz merged commit 5ac6610 into astropy:main Nov 3, 2026
12 checks passed

		assert isinstance(obj, fits.HDUList)


		def test_missions_read_product_asdf(mocker, mock_asdf_open):


		return manifest

		def read_product(self, uri, , mission=None, *kwargs):

Conversation

snbianco commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

scfleming commented May 20, 2026

Uh oh!

bsipocz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

snbianco commented Oct 3, 2026

Uh oh!

bsipocz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bsipocz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

snbianco commented May 18, 2026 •

edited

Loading

codecov Bot commented May 18, 2026 •

edited

Loading