diff --git a/EUMETSAT_data_access.ipynb b/eumetsat_metopsg/EUMETSAT_data_access.ipynb similarity index 100% rename from EUMETSAT_data_access.ipynb rename to eumetsat_metopsg/EUMETSAT_data_access.ipynb diff --git a/eumetsat_metopsg/metopsg_cloud_native.ipynb b/eumetsat_metopsg/metopsg_cloud_native.ipynb new file mode 100644 index 0000000..b153663 --- /dev/null +++ b/eumetsat_metopsg/metopsg_cloud_native.ipynb @@ -0,0 +1,2668 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "38ea46c0", + "metadata": {}, + "source": [ + "# Assessing EUMETSAT Metop-SG Products for Cloud-native Access\n", + "\n", + "\n", + "**Authors**: Rajat Shinde (UAH), Harshini Girish (UAH), Alex Mandel (Development Seed), Brian Freitag (NASA MSFC)\n", + "\n", + "**Date**: September 16, 2026\n", + "\n", + "**Description**: Metop-SGA1 carries six instrument missions: METimage (VII),\n", + "IASI-NG, MWS, a Radio Occultation sounder, 3MI, and the Copernicus\n", + "Sentinel-5/UVNS spectrometer. Their products are distributed through the\n", + "EUMETSAT Data Store as zip packages containing a data file and two XML\n", + "sidecars. This notebook checks, for one recent granule per instrument, how\n", + "close each product is to cloud-native access.\n", + "\n", + "**Setup**: This notebook will:\n", + "\n", + "1. Find the Data Store collection for each instrument\n", + "2. Check whether the hosting answers HTTP range requests\n", + "3. Open a granule lazily over HTTP with xarray, without downloading it\n", + "4. Download one granule and read its chunking and compression settings\n", + "5. Build a reference file with [VirtualiZarr](https://virtualizarr.readthedocs.io/) (and optionally write kerchunk JSON) so the remote granule can be read through the Zarr engine\n" + ] + }, + { + "cell_type": "markdown", + "id": "b9929e5b", + "metadata": {}, + "source": [ + "## Run this notebook\n", + "\n", + "You need a free EUMETSAT account. Register at\n", + "[user.eumetsat.int](https://user.eumetsat.int), then copy your consumer key\n", + "and secret from [api.eumetsat.int/api-key](https://api.eumetsat.int/api-key).\n", + "The token they produce expires after about an hour; re-run the credentials\n", + "cell if requests start returning 401." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f1e68a81", + "metadata": {}, + "outputs": [ + { + "output_type": "stream", + "text": [ + "Note: you may need to restart the kernel to use updated packages.\n" + ], + "name": "stdout" + } + ], + "source": [ + "%pip install -q eumdac \"xarray>=2024.10\" h5netcdf fsspec aiohttp kerchunk zarr pandas requests zstandard virtualizarr obstore\n" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "c733c202", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "token ok\n" + ] + } + ], + "source": [ + "import json, shutil, urllib.parse, zlib\n", + "from pathlib import Path\n", + "import numpy as np, pandas as pd, requests, fsspec, xarray as xr, eumdac\n", + "\n", + "KEY, SECRET = Path.home().joinpath(\".eumdac\", \"credentials\").read_text().strip().split(\",\")\n", + "token = eumdac.AccessToken((KEY, SECRET))\n", + "store = eumdac.DataStore(token)\n", + "\n", + "def auth():\n", + " return {\"Authorization\": f\"Bearer {token}\"} # str(token) auto-refreshes\n", + "\n", + "DATA = Path(\"data\"); DATA.mkdir(exist_ok=True)\n", + "print(\"token ok\")" + ] + }, + { + "cell_type": "markdown", + "id": "0f807adf", + "metadata": {}, + "source": [ + "## About the datasets\n", + "\n", + "The six products assessed here are the ones listed on the\n", + "[Metop-SG test data page](https://user.eumetsat.int/resources/user-guides/metop-sg-test-data):\n", + "one per instrument on Metop-SGA1.\n", + "\n", + "| Instrument | Measures | Product |\n", + "|---|---|---|\n", + "| METimage (VII) | visible and infrared radiances, 20 channels | level 1B radiances |\n", + "| MWS | microwave sounding, 24 channels | level 1B |\n", + "| Radio Occultation (GRAS-2) | GNSS bending angles | level 1B |\n", + "\n", + "They come through two different doors, and the notebook handles both:\n", + "\n", + "- **Data Store collections** exist for the instruments already distributing\n", + " flight data. As of this writing that is METimage (`EO:EUM:DAT:0464`),\n", + " MWS (`EO:EUM:DAT:0450`) and GRAS-2 radio occultation (`EO:EUM:DAT:0452`).\n", + "- **Test-data downloads** are direct links on the page above, for the\n", + " instruments not yet in the Data Store (IASI-NG, 3MI, Sentinel-5) and as\n", + " pre-launch samples for the others. Open the page in a browser, copy each\n", + " product's download link, and paste it below.\n", + "\n", + "The distinction is itself part of the assessment: a test-data link tells you\n", + "about the file format EUMETSAT intends to ship, while only a Data Store\n", + "collection tells you about the hosting the operational data will live behind.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "021bc277", + "metadata": {}, + "outputs": [], + "source": [ + "# Each product is either a Data Store collection ID or a direct download\n", + "# link copied from the Metop-SG test data page. Fill in the missing links.\n", + "PRODUCTS = {\n", + " \"METimage (VII)\": {\"collection\": \"EO:EUM:DAT:0464\"},\n", + " \"MWS\": {\"collection\": \"EO:EUM:DAT:0450\"},\n", + " \"Radio Occultation\": {\"collection\": \"EO:EUM:DAT:0452\"},\n", + "}" + ] + }, + { + "cell_type": "markdown", + "id": "76dd7a57", + "metadata": {}, + "source": [ + "## Helper functions\n", + "\n", + "Short helpers used in every section. Prefer the function docstrings below over\n", + "re-describing them here.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "999c672d", + "metadata": {}, + "outputs": [], + "source": [ + "enc = lambda s: urllib.parse.quote(s, safe=\"\")\n", + "\n", + "def latest(collection_id):\n", + " \"\"\"Newest product in a collection, plus the files in its package.\"\"\"\n", + " prod = store.get_collection(collection_id).search().first()\n", + " entries = list(prod.entries)\n", + " print(prod, \"\\n files:\", entries)\n", + " return prod, entries\n", + "\n", + "def entry_url(prod, filename):\n", + " \"\"\"Direct URL for one file inside the product, bypassing the zip.\"\"\"\n", + " return f\"{prod.url.split('?')[0]}/entry?name={enc(filename)}\"\n", + "\n", + "def ranges_ok(url):\n", + " \"\"\"True when the server answers 206 for both a leading and a suffix range.\"\"\"\n", + " codes = {}\n", + " for label, rng in ((\"head\", \"bytes=0-1023\"), (\"tail\", \"bytes=-65536\")):\n", + " r = requests.get(url, headers={**auth(), \"Range\": rng}, stream=True, timeout=60)\n", + " codes[label] = r.status_code\n", + " if label == \"head\":\n", + " print(\"first bytes:\", r.raw.read(8).hex(), \" (894844... means HDF5/netCDF-4)\")\n", + " r.close()\n", + " print(\"range status:\", codes)\n", + " return codes[\"head\"] == 206 and codes[\"tail\"] == 206\n", + "\n", + "def fetch(url, filename=None):\n", + " \"\"\"Download a direct test-data link. Zips need unzipping afterwards.\"\"\"\n", + " dest = DATA / (filename or url.split(\"/\")[-1].split(\"?\")[0])\n", + " if not dest.exists():\n", + " with requests.get(url, headers=auth(), stream=True, timeout=600) as r:\n", + " r.raise_for_status()\n", + " with open(dest, \"wb\") as f:\n", + " shutil.copyfileobj(r.raw, f)\n", + " print(f\"{dest.name}: {dest.stat().st_size/1e6:.0f} MB\")\n", + " return dest\n", + "\n", + "def remote_size(url):\n", + " \"\"\"File size via a 1-byte ranged GET, since the endpoint may not do HEAD.\"\"\"\n", + " r = requests.get(url, headers={**auth(), \"Range\": \"bytes=0-0\"},\n", + " stream=True, timeout=60)\n", + " r.close()\n", + " if r.status_code == 206:\n", + " return int(r.headers[\"Content-Range\"].split(\"/\")[-1])\n", + " return int(r.headers[\"Content-Length\"])\n", + "\n", + "def open_remote(url, log_requests=True):\n", + " \"\"\"Lazy-open a remote netCDF-4 over HTTP. Downloads nothing.\n", + "\n", + " Uses fsspec blockcache so ``block_size`` actually batches range reads.\n", + " When ``log_requests`` is True, prints how many range requests and how many\n", + " bytes the open itself required.\n", + " \"\"\"\n", + " fs = fsspec.filesystem(\"https\", client_kwargs={\"headers\": auth()},\n", + " encoded=True, skip_instance_cache=True)\n", + " f = fs.open(url, mode=\"rb\", block_size=4 * 2**20, cache_type=\"blockcache\",\n", + " size=remote_size(url))\n", + " tree = xr.open_datatree(f, engine=\"h5netcdf\", phony_dims=\"access\",\n", + " decode_times=False)\n", + " if log_requests and getattr(f, \"cache\", None) is not None:\n", + " print(f\"open_remote: {f.cache.miss_count} range requests, \"\n", + " f\"{f.cache.total_requested_bytes / 1e6:.1f} MB transferred\")\n", + " return tree\n", + "\n", + "def download(prod, filename):\n", + " \"\"\"Fetch one file from the product through the entry endpoint.\"\"\"\n", + " dest = DATA / filename\n", + " if not dest.exists():\n", + " with prod.open(entry=filename) as src, open(dest, \"wb\") as dst:\n", + " shutil.copyfileobj(src, dst)\n", + " print(f\"{dest.name}: {dest.stat().st_size/1e6:.0f} MB\")\n", + " return dest\n", + "\n", + "def layout(path):\n", + " \"\"\"Chunk shape, chunk size, and codec for every variable in the file.\"\"\"\n", + " tree = xr.open_datatree(path, engine=\"h5netcdf\", phony_dims=\"access\", decode_times=False)\n", + " rows = []\n", + " for node in tree.subtree:\n", + " for name, v in node.ds.data_vars.items():\n", + " ch = v.encoding.get(\"chunksizes\")\n", + " rows.append({\"variable\": f\"{node.path}/{name}\", \"shape\": tuple(v.shape),\n", + " \"MB\": round(v.nbytes / 1e6, 2),\n", + " \"chunks\": tuple(ch) if ch else None,\n", + " \"chunk_MB\": round(v.dtype.itemsize * int(np.prod(ch)) / 1e6, 3) if ch else None,\n", + " \"codec\": v.encoding.get(\"compression\"),\n", + " \"shuffle\": bool(v.encoding.get(\"shuffle\"))})\n", + " return pd.DataFrame(rows).sort_values(\"MB\", ascending=False)\n" + ] + }, + { + "cell_type": "markdown", + "id": "9646e466", + "metadata": {}, + "source": [ + "## METimage (VII) level 1B radiances\n", + "\n", + "The full walkthrough. Later instruments repeat the discovery → range check →\n", + "lazy open → layout steps. METimage also covers read-amplification timing and\n", + "VirtualiZarr/kerchunk references.\n", + "\n", + "First, the newest granule and its package contents. Expect one `.nc` and two\n", + "XML sidecars: the zip exists to carry those sidecars.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "184c83eb", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "W_XX-EUMETSAT-Darmstadt,SAT,SGA1-VII-1B-RAD_C_EUMT_20260922162541_G_O_20260922160859_20260922161000_C_N_T__ \n", + " files: ['W_XX-EUMETSAT-Darmstadt,SAT,SGA1-VII-1B-RAD_C_EUMT_20260922162541_G_O_20260922160859_20260922161000_C_N_T__.nc', 'EOPMetadata.xml', 'manifest.xml']\n" + ] + } + ], + "source": [ + "spec = PRODUCTS[\"METimage (VII)\"]\n", + "prod, entries = latest(spec[\"collection\"])\n", + "nc = next(e for e in entries if e.endswith(\".nc\"))\n", + "url = entry_url(prod, nc)" + ] + }, + { + "cell_type": "markdown", + "id": "5c5ca2b7", + "metadata": {}, + "source": [ + "Check range support on the per-file URL. Without 206 responses here, nothing\n", + "else in this notebook is possible and the product is download-only." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "11775c38", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "first bytes: 894844460d0a1a0a (894844... means HDF5/netCDF-4)\n", + "range status: {'head': 206, 'tail': 206}\n" + ] + }, + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "ranges_ok(url)" + ] + }, + { + "cell_type": "markdown", + "id": "c2cabe81", + "metadata": {}, + "source": [ + "Open the granule over HTTP. This reads metadata by range request and defers\n", + "everything else, so it should finish in seconds even though the file is over\n", + "100 MB. The request log from ``open_remote`` shows how chatty that metadata\n", + "pass is.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "6da1635d", + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "open_remote: 2 range requests, 8.4 MB transferred\n", + "CPU times: user 742 ms, sys: 18.8 ms, total: 761 ms\n", + "Wall time: 20.1 s\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
<xarray.DataTree>\n",
+       "Group: /\n",
+       "│   Attributes: (12/21)\n",
+       "│       title:                   VII L1B Radiances\n",
+       "│       Conventions:             CF-1.6\n",
+       "│       metadata_conventions:    Unidata Dataset Discovery v1.0\n",
+       "│       product_name:            W_XX-EUMETSAT-Darmstadt,SAT,SGA1-VII-1B-RAD_C_EU...\n",
+       "│       summary:                 VII/METimage L1B top of the atmosphere radiances\n",
+       "│       doi:                     \n",
+       "│       ...                      ...\n",
+       "│       sensing_start_time_utc:  2026-09-22 16:08:59.725\n",
+       "│       sensing_end_time_utc:    2026-09-22 16:10:00.215\n",
+       "│       environment:             Operational\n",
+       "│       references:              www.eumetsat.int\n",
+       "│       orbit_start:             5764\n",
+       "│       orbit_end:               5764\n",
+       "├── Group: /status\n",
+       "│   ├── Group: /status/satellite\n",
+       "│   │       Dimensions:                   ()\n",
+       "│   │       Data variables: (12/24)\n",
+       "│   │           epoch_time_utc            float64 8B ...\n",
+       "│   │           semi_major_axis           float64 8B ...\n",
+       "│   │           eccentricity              float64 8B ...\n",
+       "│   │           inclination               float64 8B ...\n",
+       "│   │           perigee_argument          float64 8B ...\n",
+       "│   │           right_ascension           float64 8B ...\n",
+       "│   │           ...                        ...\n",
+       "│   │           z_velocity                float64 8B ...\n",
+       "│   │           yaw_error                 float64 8B ...\n",
+       "│   │           roll_error                float64 8B ...\n",
+       "│   │           pitch_error               float64 8B ...\n",
+       "│   │           leap_second_time_utc      float64 8B ...\n",
+       "│   │           leap_second_value         float32 4B ...\n",
+       "│   ├── Group: /status/instrument\n",
+       "│   │       Dimensions:              (mode_items: 1)\n",
+       "│   │       Dimensions without coordinates: mode_items\n",
+       "│   │       Data variables:\n",
+       "│   │           mode_start_time_utc  (mode_items) float64 8B ...\n",
+       "│   │           mode_end_time_utc    (mode_items) float64 8B ...\n",
+       "│   │           instrument_mode      (mode_items) <U4 16B ...\n",
+       "│   └── Group: /status/processing\n",
+       "│           Dimensions:            ()\n",
+       "│           Data variables:\n",
+       "│               creation_time_utc  float64 8B ...\n",
+       "│           Attributes:\n",
+       "│               processor_name:              VII_L1B\n",
+       "│               processor_version:           1.0\n",
+       "│               processing_mode:             NRT\n",
+       "│               format_version:              6.0\n",
+       "│               auxiliary_data_version:      EUM/LEO-EPSSG/SPE/14/777147 v4A\n",
+       "│               pgs_reference_and_version:   EUM/LEO-EPSSG/DOC/14/746628 v5\n",
+       "│               pfs_reference_and_version:   EUM/LEO-EPSSG/SPE/14/777138 v5\n",
+       "│               atbd_reference_and_version:  EUM/LEO-EPSSG/DOC/13/702485 v4\n",
+       "│               source:                      ['SGA1_VII_1B_AUX_LMDB___S20250813000000Z_Ex...\n",
+       "├── Group: /data\n",
+       "│   ├── Group: /data/measurement_data\n",
+       "│   │       Dimensions:              (num_tie_points_alt: 140, num_tie_points_act: 394,\n",
+       "│   │                                 num_lines: 840, num_pixels: 3144, num_scans: 35)\n",
+       "│   │       Dimensions without coordinates: num_tie_points_alt, num_tie_points_act,\n",
+       "│   │                                       num_lines, num_pixels, num_scans\n",
+       "│   │       Data variables: (12/32)\n",
+       "│   │           latitude             (num_tie_points_alt, num_tie_points_act) float64 441kB ...\n",
+       "│   │           longitude            (num_tie_points_alt, num_tie_points_act) float64 441kB ...\n",
+       "│   │           delta_lat_N_dem      (num_lines, num_pixels) float32 11MB ...\n",
+       "│   │           delta_lon_E_dem      (num_lines, num_pixels) float32 11MB ...\n",
+       "│   │           solar_zenith         (num_tie_points_alt, num_tie_points_act) float64 441kB ...\n",
+       "│   │           solar_azimuth        (num_tie_points_alt, num_tie_points_act) float64 441kB ...\n",
+       "│   │           ...                   ...\n",
+       "│   │           vii_6725             (num_lines, num_pixels) float32 11MB ...\n",
+       "│   │           vii_7325             (num_lines, num_pixels) float32 11MB ...\n",
+       "│   │           vii_8540             (num_lines, num_pixels) float32 11MB ...\n",
+       "│   │           vii_10690            (num_lines, num_pixels) float32 11MB ...\n",
+       "│   │           vii_12020            (num_lines, num_pixels) float32 11MB ...\n",
+       "│   │           vii_13345            (num_lines, num_pixels) float32 11MB ...\n",
+       "│   ├── Group: /data/calibration_data\n",
+       "│   │       Dimensions:                         (isrf_samples_443: 121,\n",
+       "│   │                                            isrf_samples_555: 81,\n",
+       "│   │                                            isrf_samples_668: 81,\n",
+       "│   │                                            isrf_samples_752: 71,\n",
+       "│   │                                            isrf_samples_763: 71,\n",
+       "│   │                                            isrf_samples_865: 121,\n",
+       "│   │                                            ...\n",
+       "│   │                                            isrf_samples_7325: 160,\n",
+       "│   │                                            isrf_samples_8540: 125,\n",
+       "│   │                                            isrf_samples_10690: 124,\n",
+       "│   │                                            isrf_samples_12020: 69,\n",
+       "│   │                                            isrf_samples_13345: 51,\n",
+       "│   │                                            num_chan_solar: 11, num_chan_thermal: 9)\n",
+       "│   │       Dimensions without coordinates: isrf_samples_443, isrf_samples_555,\n",
+       "│   │                                       isrf_samples_668, isrf_samples_752,\n",
+       "│   │                                       isrf_samples_763, isrf_samples_865,\n",
+       "│   │                                       isrf_samples_914, isrf_samples_1240,\n",
+       "│   │                                       isrf_samples_1375, isrf_samples_1630,\n",
+       "│   │                                       isrf_samples_2250, isrf_samples_3740,\n",
+       "│   │                                       isrf_samples_3959, isrf_samples_4050,\n",
+       "│   │                                       isrf_samples_6725, isrf_samples_7325,\n",
+       "│   │                                       isrf_samples_8540, isrf_samples_10690,\n",
+       "│   │                                       isrf_samples_12020, isrf_samples_13345,\n",
+       "│   │                                       num_chan_solar, num_chan_thermal\n",
+       "│   │       Data variables: (12/44)\n",
+       "│   │           ISRF_443                        (isrf_samples_443) float32 484B ...\n",
+       "│   │           ISRF_443_wavelength             (isrf_samples_443) float64 968B ...\n",
+       "│   │           ISRF_555                        (isrf_samples_555) float32 324B ...\n",
+       "│   │           ISRF_555_wavelength             (isrf_samples_555) float64 648B ...\n",
+       "│   │           ISRF_668                        (isrf_samples_668) float32 324B ...\n",
+       "│   │           ISRF_668_wavelength             (isrf_samples_668) float64 648B ...\n",
+       "│   │           ...                              ...\n",
+       "│   │           ISRF_13345                      (isrf_samples_13345) float32 204B ...\n",
+       "│   │           ISRF_13345_wavelength           (isrf_samples_13345) float64 408B ...\n",
+       "│   │           band_averaged_solar_irradiance  (num_chan_solar) float32 44B ...\n",
+       "│   │           channel_cw_thermal              (num_chan_thermal) float32 36B ...\n",
+       "│   │           bt_conversion_a                 (num_chan_thermal) float32 36B ...\n",
+       "│   │           bt_conversion_b                 (num_chan_thermal) float32 36B ...\n",
+       "│   ├── Group: /data/quality_information\n",
+       "│   │       Dimensions:              (num_scans: 35, num_chan: 20)\n",
+       "│   │       Dimensions without coordinates: num_scans, num_chan\n",
+       "│   │       Data variables:\n",
+       "│   │           data_quality         (num_scans, num_chan) uint8 700B ...\n",
+       "│   │           calibration_quality  (num_scans, num_chan) uint16 1kB ...\n",
+       "│   │           geolocation_quality  (num_scans) uint8 35B ...\n",
+       "│   └── Group: /data/processing_flags\n",
+       "│           Dimensions:                 (num_scans: 35, num_chan: 20, num_pixels_alt: 24,\n",
+       "│                                        num_pixels: 3144)\n",
+       "│           Dimensions without coordinates: num_scans, num_chan, num_pixels_alt, num_pixels\n",
+       "│           Data variables:\n",
+       "│               vii_processing_flag     (num_scans, num_chan) uint8 700B ...\n",
+       "│               pixel_duplication_mask  (num_pixels_alt, num_pixels) uint8 75kB ...\n",
+       "│               pixel_performance_mask  (num_chan, num_pixels_alt) uint8 480B ...\n",
+       "│               processing_mode         uint8 1B ...\n",
+       "└── Group: /quality\n",
+       "        Dimensions:                    ()\n",
+       "        Data variables:\n",
+       "            duration_of_product        float64 8B ...\n",
+       "            duration_of_data_present   float64 8B ...\n",
+       "            duration_of_data_missing   float64 8B ...\n",
+       "            duration_of_data_degraded  float64 8B ...\n",
+       "        Attributes:\n",
+       "            overall_quality_flag:  0
" + ], + "text/plain": [ + "\n", + "Group: /\n", + "│ Attributes: (12/21)\n", + "│ title: VII L1B Radiances\n", + "│ Conventions: CF-1.6\n", + "│ metadata_conventions: Unidata Dataset Discovery v1.0\n", + "│ product_name: W_XX-EUMETSAT-Darmstadt,SAT,SGA1-VII-1B-RAD_C_EU...\n", + "│ summary: VII/METimage L1B top of the atmosphere radiances\n", + "│ doi: \n", + "│ ... ...\n", + "│ sensing_start_time_utc: 2026-09-22 16:08:59.725\n", + "│ sensing_end_time_utc: 2026-09-22 16:10:00.215\n", + "│ environment: Operational\n", + "│ references: www.eumetsat.int\n", + "│ orbit_start: 5764\n", + "│ orbit_end: 5764\n", + "├── Group: /status\n", + "│ ├── Group: /status/satellite\n", + "│ │ Dimensions: ()\n", + "│ │ Data variables: (12/24)\n", + "│ │ epoch_time_utc float64 8B ...\n", + "│ │ semi_major_axis float64 8B ...\n", + "│ │ eccentricity float64 8B ...\n", + "│ │ inclination float64 8B ...\n", + "│ │ perigee_argument float64 8B ...\n", + "│ │ right_ascension float64 8B ...\n", + "│ │ ... ...\n", + "│ │ z_velocity float64 8B ...\n", + "│ │ yaw_error float64 8B ...\n", + "│ │ roll_error float64 8B ...\n", + "│ │ pitch_error float64 8B ...\n", + "│ │ leap_second_time_utc float64 8B ...\n", + "│ │ leap_second_value float32 4B ...\n", + "│ ├── Group: /status/instrument\n", + "│ │ Dimensions: (mode_items: 1)\n", + "│ │ Dimensions without coordinates: mode_items\n", + "│ │ Data variables:\n", + "│ │ mode_start_time_utc (mode_items) float64 8B ...\n", + "│ │ mode_end_time_utc (mode_items) float64 8B ...\n", + "│ │ instrument_mode (mode_items) \n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
variableshapeMBchunkschunk_MBcodecshuffle
46/data/measurement_data/vii_668(840, 3144)10.56(1, 3144)0.013NoneFalse
45/data/measurement_data/vii_555(840, 3144)10.56(1, 3144)0.013NoneFalse
34/data/measurement_data/delta_lat_N_dem(840, 3144)10.56(1, 3144)0.013NoneFalse
35/data/measurement_data/delta_lon_E_dem(840, 3144)10.56(1, 3144)0.013NoneFalse
50/data/measurement_data/vii_914(840, 3144)10.56(1, 3144)0.013NoneFalse
48/data/measurement_data/vii_763(840, 3144)10.56(1, 3144)0.013NoneFalse
49/data/measurement_data/vii_865(840, 3144)10.56(1, 3144)0.013NoneFalse
47/data/measurement_data/vii_752(840, 3144)10.56(1, 3144)0.013NoneFalse
44/data/measurement_data/vii_443(840, 3144)10.56(1, 3144)0.013NoneFalse
60/data/measurement_data/vii_8540(840, 3144)10.56(1, 3144)0.013NoneFalse
59/data/measurement_data/vii_7325(840, 3144)10.56(1, 3144)0.013NoneFalse
58/data/measurement_data/vii_6725(840, 3144)10.56(1, 3144)0.013NoneFalse
57/data/measurement_data/vii_4050(840, 3144)10.56(1, 3144)0.013NoneFalse
56/data/measurement_data/vii_3959(840, 3144)10.56(1, 3144)0.013NoneFalse
55/data/measurement_data/vii_3740(840, 3144)10.56(1, 3144)0.013NoneFalse
\n", + "" + ], + "text/plain": [ + " variable shape MB chunks \\\n", + "46 /data/measurement_data/vii_668 (840, 3144) 10.56 (1, 3144) \n", + "45 /data/measurement_data/vii_555 (840, 3144) 10.56 (1, 3144) \n", + "34 /data/measurement_data/delta_lat_N_dem (840, 3144) 10.56 (1, 3144) \n", + "35 /data/measurement_data/delta_lon_E_dem (840, 3144) 10.56 (1, 3144) \n", + "50 /data/measurement_data/vii_914 (840, 3144) 10.56 (1, 3144) \n", + "48 /data/measurement_data/vii_763 (840, 3144) 10.56 (1, 3144) \n", + "49 /data/measurement_data/vii_865 (840, 3144) 10.56 (1, 3144) \n", + "47 /data/measurement_data/vii_752 (840, 3144) 10.56 (1, 3144) \n", + "44 /data/measurement_data/vii_443 (840, 3144) 10.56 (1, 3144) \n", + "60 /data/measurement_data/vii_8540 (840, 3144) 10.56 (1, 3144) \n", + "59 /data/measurement_data/vii_7325 (840, 3144) 10.56 (1, 3144) \n", + "58 /data/measurement_data/vii_6725 (840, 3144) 10.56 (1, 3144) \n", + "57 /data/measurement_data/vii_4050 (840, 3144) 10.56 (1, 3144) \n", + "56 /data/measurement_data/vii_3959 (840, 3144) 10.56 (1, 3144) \n", + "55 /data/measurement_data/vii_3740 (840, 3144) 10.56 (1, 3144) \n", + "\n", + " chunk_MB codec shuffle \n", + "46 0.013 None False \n", + "45 0.013 None False \n", + "34 0.013 None False \n", + "35 0.013 None False \n", + "50 0.013 None False \n", + "48 0.013 None False \n", + "49 0.013 None False \n", + "47 0.013 None False \n", + "44 0.013 None False \n", + "60 0.013 None False \n", + "59 0.013 None False \n", + "58 0.013 None False \n", + "57 0.013 None False \n", + "56 0.013 None False \n", + "55 0.013 None False " + ] + }, + "execution_count": 23, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "local = download(prod, nc)\n", + "L = layout(local)\n", + "L.head(15)" + ] + }, + { + "cell_type": "markdown", + "id": "7dd6a11e", + "metadata": {}, + "source": [ + "Summarize the layout table. `nbytes` / in-memory size is the sum of each\n", + "array's `dtype.itemsize * nelements` as Python would hold it uncompressed.\n", + "That is not always identical to an HDF5 \"logical size,\" but it is the right\n", + "baseline to compare against the on-disk file size. `on disk` is the actual\n", + "file size. Tiny median `chunk_MB` values mean remote subsetting will issue\n", + "many small range requests.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "92435aab", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "compressed variables: 0 of 115\n", + "in-memory size (sum of nbytes): 235 MB on disk: 119 MB\n", + "chunk sizes (MB): {'min': 0.0, '50%': 0.013, 'max': 0.013}\n" + ] + } + ], + "source": [ + "print(\"compressed variables:\", int(L.codec.notna().sum()), \"of\", len(L))\n", + "print(\"in-memory size (sum of nbytes):\", round(L.MB.sum()), \"MB on disk:\",\n", + " round(local.stat().st_size / 1e6), \"MB\")\n", + "print(\"chunk sizes (MB):\", L.chunk_MB.describe()[[\"min\", \"50%\", \"max\"]].round(3).to_dict())\n" + ] + }, + { + "cell_type": "markdown", + "id": "df387104", + "metadata": {}, + "source": [ + "Measure what internal compression would buy. This compresses **one on-disk\n", + "chunk** on local CPU, with and without an HDF5-style byte shuffle (bytes of\n", + "each element grouped by significance).\n", + "\n", + "On many float fields, deflate level 1 with shuffle is the cheap, widely\n", + "readable option and zstd does the same job faster. Ratios above about 1.5×\n", + "make a solid case. If shuffle *hurts* on a given chunk, that often means the\n", + "native byte layout already has long runs (fills, constants) that shuffle\n", + "scatters — report the better of the two rather than assuming shuffle always wins.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "7697e069", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "/data/measurement_data/vii_668 (1, 3144) float32 chunk (1, 3144)\n", + "deflate-1: 141.30x, with shuffle 149.71x\n", + "zstd-3: 571.64x, with shuffle 465.78x\n" + ] + } + ], + "source": [ + "import zstandard as zstd\n", + "\n", + "def hdf5_byte_shuffle(arr):\n", + " \"\"\"HDF5 shuffle: group byte-0 of every element, then byte-1, ...\"\"\"\n", + " a = np.ascontiguousarray(arr)\n", + " return a.view(np.uint8).reshape(-1, a.dtype.itemsize).ravel(order=\"F\").tobytes()\n", + "\n", + "row = L.dropna(subset=[\"chunks\"]).iloc[0]\n", + "node = tree[str(Path(row.variable).parent)].ds\n", + "name = Path(row.variable).name\n", + "# one storage chunk, not an arbitrary 512² window\n", + "slices = tuple(slice(0, c) for c in row.chunks)\n", + "arr = np.ascontiguousarray(node[name][slices].values)\n", + "raw = arr.tobytes()\n", + "shuf = hdf5_byte_shuffle(arr)\n", + "\n", + "print(row.variable, arr.shape, arr.dtype, \"chunk\", row.chunks)\n", + "print(f\"deflate-1: {len(raw)/len(zlib.compress(raw, 1)):.2f}x, \"\n", + " f\"with shuffle {len(raw)/len(zlib.compress(shuf, 1)):.2f}x\")\n", + "c = zstd.ZstdCompressor(level=3)\n", + "print(f\"zstd-3: {len(raw)/len(c.compress(raw)):.2f}x, \"\n", + " f\"with shuffle {len(raw)/len(c.compress(shuf)):.2f}x\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "5269ed83-aff3-4002-8a95-2dcde92392e5", + "metadata": {}, + "source": [ + "### Read efficiency and virtual references\n", + "\n", + "Still on the METimage granule from above: compare a local slice to the same\n", + "slice over HTTP, counting range requests, then build a VirtualiZarr reference\n", + "file (kerchunk JSON) so subsequent opens skip the expensive HDF5 metadata walk.\n", + "\n", + "These cells reuse `url`, `local`, `auth`, and `remote_size` from the helper\n", + "section rather than hard-coding a filename.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "a2a610ee-4730-4e4f-a846-620362dcc977", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "variable : vii_443 (840, 3144) float32\n", + "chunks : (1, 3144) codec=None\n", + "file : 119 MB\n", + "\n", + "local slice : 16 KB in 710 ms\n", + "\n", + "open : 8.4 MB in 2 reads, 7.2s\n", + "slice : 4.2 MB in 1 reads, 1.8s -> 256x amplification for 16 KB wanted\n", + "total : 12.6 MB of a 119 MB file\n" + ] + } + ], + "source": [ + "import time\n", + "\n", + "# Target mid-swath so we are not reading a corner that happens to be fill.\n", + "rad = tree[\"data/measurement_data\"].ds\n", + "var = next(v for v in rad.data_vars if v.startswith(\"vii_\"))\n", + "ny, nx = rad[var].shape[:2]\n", + "sl = (slice(ny // 2, ny // 2 + 64), slice(nx // 2, nx // 2 + 64))\n", + "chunks = rad[var].encoding.get(\"chunksizes\")\n", + "print(f\"variable : {var} {rad[var].shape} {rad[var].dtype}\")\n", + "print(f\"chunks : {chunks} codec={rad[var].encoding.get('compression')}\")\n", + "print(f\"file : {local.stat().st_size / 1e6:.0f} MB\\n\")\n", + "\n", + "# ---- local baseline -------------------------------------------------------\n", + "t0 = time.perf_counter()\n", + "a = xr.open_datatree(local, engine=\"h5netcdf\", phony_dims=\"access\",\n", + " decode_times=False)[\"data/measurement_data\"].ds[var][sl].values\n", + "t_local = time.perf_counter() - t0\n", + "wanted = a.nbytes\n", + "print(f\"local slice : {wanted / 1e3:.0f} KB in {t_local * 1e3:.0f} ms\\n\")\n", + "\n", + "# ---- remote: reopen with blockcache so miss_count is meaningful ----------\n", + "bs = 4 * 2**20\n", + "fs = fsspec.filesystem(\"https\", client_kwargs={\"headers\": auth()},\n", + " encoded=True, skip_instance_cache=True)\n", + "f = fs.open(url, mode=\"rb\", block_size=bs, cache_type=\"blockcache\",\n", + " size=remote_size(url))\n", + "\n", + "t0 = time.perf_counter()\n", + "rt = xr.open_datatree(f, engine=\"h5netcdf\", phony_dims=\"access\", decode_times=False)\n", + "t_open = time.perf_counter() - t0\n", + "open_bytes, open_reqs = f.cache.total_requested_bytes, f.cache.miss_count\n", + "\n", + "t0 = time.perf_counter()\n", + "b = rt[\"data/measurement_data\"].ds[var][sl].values\n", + "t_read = time.perf_counter() - t0\n", + "read_bytes = f.cache.total_requested_bytes - open_bytes\n", + "read_reqs = f.cache.miss_count - open_reqs\n", + "\n", + "print(f\"open : {open_bytes / 1e6:6.1f} MB in {open_reqs:3d} reads, {t_open:5.1f}s\")\n", + "print(f\"slice : {read_bytes / 1e6:6.1f} MB in {read_reqs:3d} reads, {t_read:5.1f}s\"\n", + " f\" -> {read_bytes / wanted:.0f}x amplification for {wanted / 1e3:.0f} KB wanted\")\n", + "print(f\"total : {(open_bytes + read_bytes) / 1e6:6.1f} MB of a \"\n", + " f\"{local.stat().st_size / 1e6:.0f} MB file\")\n", + "assert np.allclose(a, b, equal_nan=True)\n" + ] + }, + { + "cell_type": "markdown", + "id": "f65f8430-3766-4f7f-8dde-316d106e2038", + "metadata": {}, + "source": [ + "### VirtualiZarr → kerchunk references\n", + "\n", + "VirtualiZarr reads the local netCDF once and writes a small reference file.\n", + "Readers then use that sidecar (kerchunk JSON) to fetch only the bytes they\n", + "need from the remote URL — without re-walking HDF5 metadata on every open.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "89f82bc0-87a6-4637-9973-e90a83ce8db0", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "data/metimage_l1b_sample.nc 119 MB\n", + "remote url: https://api.eumetsat.int/data/download/1.0.0/collections/EO%3AEUM%3ADAT%3A0464/products/W_XX-EUMETSAT-Darmstadt%2CSAT%2CSGA1-VII-1B-RAD_C_EUMT_20260922162541_G_O_20260922160859_20260922161000_C_N_T__/entry?name=W_XX-EUMETSAT-Darmstadt%2CSAT%2CSGA1-VII-1B-RAD_C_EUMT_20260922162541_G_O_20260922160859_20260922161000_C_N_T__.nc\n" + ] + } + ], + "source": [ + "# Index the local download, then rename chunk paths to the remote Data Store URL.\n", + "sample = DATA / \"metimage_l1b_sample.nc\"\n", + "if not sample.exists():\n", + " shutil.copy2(local, sample)\n", + "print(sample, f\"{sample.stat().st_size / 1e6:.0f} MB\")\n", + "print(\"remote url:\", url)\n" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "f9abbfdc-047e-4c39-a665-d0465964ea76", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
<xarray.Dataset> Size: 118MB\n",
+       "Dimensions:              (num_tie_points_alt: 140, num_tie_points_act: 394,\n",
+       "                          data/num_lines: 840, data/num_pixels: 3144,\n",
+       "                          data/num_scans: 35)\n",
+       "Dimensions without coordinates: num_tie_points_alt, num_tie_points_act,\n",
+       "                                data/num_lines, data/num_pixels, data/num_scans\n",
+       "Data variables: (12/32)\n",
+       "    latitude             (num_tie_points_alt, num_tie_points_act) int32 221kB ManifestArray<shape=(140, 394), dtype=int32, chunks=(1, 394)...\n",
+       "    longitude            (num_tie_points_alt, num_tie_points_act) uint32 221kB ManifestArray<shape=(140, 394), dtype=uint32, chunks=(1, 394...\n",
+       "    delta_lat_N_dem      (data/num_lines, data/num_pixels) int16 5MB Manifest...\n",
+       "    delta_lon_E_dem      (data/num_lines, data/num_pixels) int16 5MB Manifest...\n",
+       "    solar_zenith         (num_tie_points_alt, num_tie_points_act) uint32 221kB ManifestArray<shape=(140, 394), dtype=uint32, chunks=(1, 394...\n",
+       "    solar_azimuth        (num_tie_points_alt, num_tie_points_act) uint32 221kB ManifestArray<shape=(140, 394), dtype=uint32, chunks=(1, 394...\n",
+       "    ...                   ...\n",
+       "    vii_6725             (data/num_lines, data/num_pixels) uint16 5MB Manifes...\n",
+       "    vii_7325             (data/num_lines, data/num_pixels) uint16 5MB Manifes...\n",
+       "    vii_8540             (data/num_lines, data/num_pixels) uint16 5MB Manifes...\n",
+       "    vii_10690            (data/num_lines, data/num_pixels) uint16 5MB Manifes...\n",
+       "    vii_12020            (data/num_lines, data/num_pixels) uint16 5MB Manifes...\n",
+       "    vii_13345            (data/num_lines, data/num_pixels) uint16 5MB Manifes...
" + ], + "text/plain": [ + " Size: 118MB\n", + "Dimensions: (num_tie_points_alt: 140, num_tie_points_act: 394,\n", + " data/num_lines: 840, data/num_pixels: 3144,\n", + " data/num_scans: 35)\n", + "Dimensions without coordinates: num_tie_points_alt, num_tie_points_act,\n", + " data/num_lines, data/num_pixels, data/num_scans\n", + "Data variables: (12/32)\n", + " latitude (num_tie_points_alt, num_tie_points_act) int32 221kB ManifestArray\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
variableshapeMBchunkschunk_MBcodecshuffle
130/data/calibration/mws_toa_radiance(79, 95, 24)1.44(28, 95, 24)0.511NoneFalse
108/data/calibration/mws_toa_brightness_temperature(79, 95, 24)1.44(28, 95, 24)0.511NoneFalse
136/data/measurement/mws_earth_view_counts(79, 95, 24)0.72(57, 95, 24)0.520NoneFalse
141/data/processing_information/mws_radiance_flag(79, 95, 24)0.18(79, 95, 24)0.180NoneFalse
162/data/processing_information/mws_brightnesstem...(79, 95, 24)0.18(79, 95, 24)0.180NoneFalse
99/data/navigation/mws_surface_type(79, 95, 2)0.12NoneNaNNoneFalse
96/data/navigation/mws_solar_azimuth_angle(79, 95)0.06NoneNaNNoneFalse
95/data/navigation/mws_satellite_zenith_angle(79, 95)0.06NoneNaNNoneFalse
135/data/measurement/mws_earth_view_counts_os_stdev(79, 95, 2)0.06NoneNaNNoneFalse
94/data/navigation/mws_solar_zenith_angle(79, 95)0.06NoneNaNNoneFalse
\n", + "" + ], + "text/plain": [ + " variable shape MB \\\n", + "130 /data/calibration/mws_toa_radiance (79, 95, 24) 1.44 \n", + "108 /data/calibration/mws_toa_brightness_temperature (79, 95, 24) 1.44 \n", + "136 /data/measurement/mws_earth_view_counts (79, 95, 24) 0.72 \n", + "141 /data/processing_information/mws_radiance_flag (79, 95, 24) 0.18 \n", + "162 /data/processing_information/mws_brightnesstem... (79, 95, 24) 0.18 \n", + "99 /data/navigation/mws_surface_type (79, 95, 2) 0.12 \n", + "96 /data/navigation/mws_solar_azimuth_angle (79, 95) 0.06 \n", + "95 /data/navigation/mws_satellite_zenith_angle (79, 95) 0.06 \n", + "135 /data/measurement/mws_earth_view_counts_os_stdev (79, 95, 2) 0.06 \n", + "94 /data/navigation/mws_solar_zenith_angle (79, 95) 0.06 \n", + "\n", + " chunks chunk_MB codec shuffle \n", + "130 (28, 95, 24) 0.511 None False \n", + "108 (28, 95, 24) 0.511 None False \n", + "136 (57, 95, 24) 0.520 None False \n", + "141 (79, 95, 24) 0.180 None False \n", + "162 (79, 95, 24) 0.180 None False \n", + "99 None NaN None False \n", + "96 None NaN None False \n", + "95 None NaN None False \n", + "135 None NaN None False \n", + "94 None NaN None False " + ] + }, + "execution_count": 31, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "tree = open_remote(url)\n", + "local = download(prod, datafile) if spec.get(\"collection\") else fetch(url)\n", + "L = layout(local)\n", + "print(\"compressed:\", int(L.codec.notna().sum()), \"of\", len(L),\n", + " \" median chunk MB:\", L.chunk_MB.median())\n", + "L.head(10)" + ] + }, + { + "cell_type": "markdown", + "id": "ad377891", + "metadata": {}, + "source": [ + "## Radio Occultation\n", + "\n", + "GRAS-2 has a Data Store collection, and its title says netCDF. The\n", + "`ranges_ok` call prints the first bytes as a check anyway: `894844` opens\n", + "like the others, `425546` spells BUFR, which has no internal chunking to\n", + "assess and would end this section early as its own finding.\n", + "\n", + "If the file is netCDF-4, continue with the same `open_remote` / `download` /\n", + "`layout` pattern used for MWS.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "b36ca262", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "W_XX-EUMETSAT-Darmstadt,SAT,SGA1-RO_-1B-BND_C_EUMT_20260922164656_G_O_20260922155827_20260922160416_O_N_C20 \n", + " files: ['W_XX-EUMETSAT-Darmstadt,SAT,SGA1-RO_-1B-BND_C_EUMT_20260922164656_G_O_20260922155827_20260922160416_O_N_C20.nc', 'EOPMetadata.xml', 'manifest.xml']\n", + "first bytes: 894844460d0a1a0a (894844... means HDF5/netCDF-4)\n", + "range status: {'head': 206, 'tail': 206}\n" + ] + } + ], + "source": [ + "spec = PRODUCTS[\"Radio Occultation\"]\n", + "if spec.get(\"collection\"):\n", + " prod, entries = latest(spec[\"collection\"])\n", + " datafile = next(e for e in entries if not e.lower().endswith((\".xml\", \".txt\")))\n", + " url = entry_url(prod, datafile)\n", + "else:\n", + " url = spec[\"url\"] # direct test-data link\n", + " assert url, \"paste this product's link from the test-data page into PRODUCTS\"\n", + "\n", + "ok = ranges_ok(url)\n", + "if not ok:\n", + " print(\"range requests not supported — download-only for this product\")\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python [conda env:notebook] *", + "language": "python", + "name": "conda-env-notebook-py" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}