Skip to content

Add parquet support to astroquery.mast.missions.get_product() - #3702

Open
sedonaprice wants to merge 5 commits into
astropy:mainfrom
sedonaprice:missions-cloud-uris-add-parquet-support
Open

sedonaprice wants to merge 5 commits into
astropy:mainfrom
sedonaprice:missions-cloud-uris-add-parquet-support

Conversation

@sedonaprice

Copy link
Copy Markdown

Add parquet support to the MastMissions class that can read products into memory, to support streaming of parquet files with fsspec, pandas, and pyarrow (astropy.table.Table uses pandas + pyarrow to ready parquet). As with ASDF files, Roman parquet data are redirected to S3, and so these files are not downloaded and are instead streamed directly into astropy.table.Table tables.

Streaming parquet files from S3 (similar to streaming ASDF files from S3) would be important for Roman users to access cloud-hosted catalog files.

This requires new package dependencies: pandas, pyarrow.

(Tagging @snbianco for visibility!)

Requires new optional dependencies: pandas, pyarrow.
Also added fsspec[http] to the all deps, to ensure coverage (may be covered by roman_datamodels, but covering all bases.)
@codecov

codecov Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.59259% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.06%. Comparing base (7f2fd33) to head (85e635d).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
astroquery/mast/missions.py 92.59% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3702      +/-   ##
==========================================
+ Coverage   74.04%   74.06%   +0.01%     
==========================================
  Files         228      228              
  Lines       21649    21672      +23     
==========================================
+ Hits        16031    16052      +21     
- Misses       5618     5620       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@snbianco snbianco left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this PR!

Comment thread astroquery/mast/missions.py Resolved
Comment thread astroquery/mast/missions.py Resolved
Comment thread astroquery/mast/missions.py Resolved
Comment thread pyproject.toml Resolved
Comment thread astroquery/mast/tests/test_mast.py Resolved
Comment thread astroquery/mast/tests/test_mast.py Resolved
@bsipocz bsipocz added this to the 0.4.13 milestone Nov 5, 2026
sedonaprice and others added 4 commits November 5, 2026 09:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants