From c1fb622c5b81488a4afd76ecf6e4f6aabe482983 Mon Sep 17 00:00:00 2001 From: omshinde Date: Mon, 21 Sep 2026 16:15:04 +0000 Subject: [PATCH 1/5] Add EUMETSAT examples --- .../EUMETSAT_data_access.ipynb | 0 eumetsat_metopsg/metopsg_cloud_native.ipynb | 1938 +++++++++++++++++ 2 files changed, 1938 insertions(+) rename EUMETSAT_data_access.ipynb => eumetsat_metopsg/EUMETSAT_data_access.ipynb (100%) create mode 100644 eumetsat_metopsg/metopsg_cloud_native.ipynb diff --git a/EUMETSAT_data_access.ipynb b/eumetsat_metopsg/EUMETSAT_data_access.ipynb similarity index 100% rename from EUMETSAT_data_access.ipynb rename to eumetsat_metopsg/EUMETSAT_data_access.ipynb diff --git a/eumetsat_metopsg/metopsg_cloud_native.ipynb b/eumetsat_metopsg/metopsg_cloud_native.ipynb new file mode 100644 index 0000000..43bafbc --- /dev/null +++ b/eumetsat_metopsg/metopsg_cloud_native.ipynb @@ -0,0 +1,1938 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "38ea46c0", + "metadata": {}, + "source": [ + "# Assessing EUMETSAT Metop-SG Products for Cloud-native Access\n", + "\n", + "\n", + "**Authors**: Rajat Shinde (UAH), Harshini Girish (UAH), Alex Mandel (Development Seed), Brian Freitag (NASA MSFC)\n", + "\n", + "**Date**: September 16, 2026\n", + "\n", + "**Description**: Metop-SGA1 carries six instrument missions: METimage (VII),\n", + "IASI-NG, MWS, a Radio Occultation sounder, 3MI, and the Copernicus\n", + "Sentinel-5/UVNS spectrometer. Their products are distributed through the\n", + "EUMETSAT Data Store as zip packages containing a data file and two XML\n", + "sidecars. This notebook checks, for one recent granule per instrument, how\n", + "close each product is to cloud-native access.\n", + "\n", + "**Setup**: This notebook will:\n", + "\n", + "1. Find the Data Store collection for each instrument\n", + "2. Check whether the hosting answers HTTP range requests\n", + "3. Open a granule lazily over HTTP with xarray, without downloading it\n", + "4. Download one granule and read its chunking and compression settings\n", + "5. Build a kerchunk index and read the remote file through the Zarr engine" + ] + }, + { + "cell_type": "markdown", + "id": "b9929e5b", + "metadata": {}, + "source": [ + "## Run this notebook\n", + "\n", + "You need a free EUMETSAT account. Register at\n", + "[user.eumetsat.int](https://user.eumetsat.int), then copy your consumer key\n", + "and secret from [api.eumetsat.int/api-key](https://api.eumetsat.int/api-key).\n", + "The token they produce expires after about an hour; re-run the credentials\n", + "cell if requests start returning 401." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "f1e68a81", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], + "source": [ + "%pip install -q eumdac \"xarray>=2024.10\" h5netcdf fsspec aiohttp kerchunk zarr pandas requests zstandard" + ] + }, + { + "cell_type": "code", + "execution_count": 44, + "id": "c733c202", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "token ok\n" + ] + } + ], + "source": [ + "import json, shutil, urllib.parse, zlib\n", + "from pathlib import Path\n", + "import numpy as np, pandas as pd, requests, fsspec, xarray as xr, eumdac\n", + "\n", + "KEY, SECRET = Path.home().joinpath(\".eumdac\", \"credentials\").read_text().strip().split(\",\")\n", + "token = eumdac.AccessToken((KEY, SECRET))\n", + "store = eumdac.DataStore(token)\n", + "\n", + "def auth():\n", + " return {\"Authorization\": f\"Bearer {token}\"} # str(token) auto-refreshes\n", + "\n", + "DATA = Path(\"data\"); DATA.mkdir(exist_ok=True)\n", + "print(\"token ok\")" + ] + }, + { + "cell_type": "markdown", + "id": "0f807adf", + "metadata": {}, + "source": [ + "## About the datasets\n", + "\n", + "The six products assessed here are the ones listed on the\n", + "[Metop-SG test data page](https://user.eumetsat.int/resources/user-guides/metop-sg-test-data):\n", + "one per instrument on Metop-SGA1.\n", + "\n", + "| Instrument | Measures | Product |\n", + "|---|---|---|\n", + "| METimage (VII) | visible and infrared radiances, 20 channels | level 1B radiances |\n", + "| MWS | microwave sounding, 24 channels | level 1B |\n", + "| Radio Occultation (GRAS-2) | GNSS bending angles | level 1B |\n", + "\n", + "They come through two different doors, and the notebook handles both:\n", + "\n", + "- **Data Store collections** exist for the instruments already distributing\n", + " flight data. As of this writing that is METimage (`EO:EUM:DAT:0464`),\n", + " MWS (`EO:EUM:DAT:0450`) and GRAS-2 radio occultation (`EO:EUM:DAT:0452`).\n", + "- **Test-data downloads** are direct links on the page above, for the\n", + " instruments not yet in the Data Store (IASI-NG, 3MI, Sentinel-5) and as\n", + " pre-launch samples for the others. Open the page in a browser, copy each\n", + " product's download link, and paste it below.\n", + "\n", + "The distinction is itself part of the assessment: a test-data link tells you\n", + "about the file format EUMETSAT intends to ship, while only a Data Store\n", + "collection tells you about the hosting the operational data will live behind.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "021bc277", + "metadata": {}, + "outputs": [], + "source": [ + "# Each product is either a Data Store collection ID or a direct download\n", + "# link copied from the Metop-SG test data page. Fill in the missing links.\n", + "PRODUCTS = {\n", + " \"METimage (VII)\": {\"collection\": \"EO:EUM:DAT:0464\"},\n", + " \"MWS\": {\"collection\": \"EO:EUM:DAT:0450\"},\n", + " \"Radio Occultation\": {\"collection\": \"EO:EUM:DAT:0452\"},\n", + "}" + ] + }, + { + "cell_type": "markdown", + "id": "76dd7a57", + "metadata": {}, + "source": [ + "## Helper functions\n", + "\n", + "Five short functions used in every section. `latest` fetches the newest\n", + "product in a collection and lists the files inside its delivery package.\n", + "`ranges_ok` sends two ranged GETs; the suffix range matters because zip\n", + "central directories and Parquet footers sit at the end of the file, so a\n", + "server that cannot serve file tails cannot serve those formats in place.\n", + "`open_remote` opens a URL with xarray through fsspec, reading only the bytes\n", + "h5netcdf asks for. `layout` tabulates chunk shape and codec per variable.\n", + "`fetch` downloads a direct test-data link. `try_kerchunk` indexes the chunks of a local copy, points the references at\n", + "the remote URL, and reads one slice back through the Zarr engine." + ] + }, + { + "cell_type": "code", + "execution_count": 47, + "id": "999c672d", + "metadata": {}, + "outputs": [], + "source": [ + "enc = lambda s: urllib.parse.quote(s, safe=\"\")\n", + "\n", + "def latest(collection_id):\n", + " \"\"\"Newest product in a collection, plus the files in its package.\"\"\"\n", + " prod = store.get_collection(collection_id).search().first()\n", + " entries = list(prod.entries)\n", + " print(prod, \"\\n files:\", entries)\n", + " return prod, entries\n", + "\n", + "def entry_url(prod, filename):\n", + " \"\"\"Direct URL for one file inside the product, bypassing the zip.\"\"\"\n", + " return f\"{prod.url.split('?')[0]}/entry?name={enc(filename)}\"\n", + "\n", + "def ranges_ok(url):\n", + " \"\"\"True when the server answers 206 for both a leading and a suffix range.\"\"\"\n", + " codes = {}\n", + " for label, rng in ((\"head\", \"bytes=0-1023\"), (\"tail\", \"bytes=-65536\")):\n", + " r = requests.get(url, headers={**auth(), \"Range\": rng}, stream=True, timeout=60)\n", + " codes[label] = r.status_code\n", + " if label == \"head\":\n", + " print(\"first bytes:\", r.raw.read(8).hex(), \" (894844... means HDF5/netCDF-4)\")\n", + " r.close()\n", + " print(\"range status:\", codes)\n", + " return codes[\"head\"] == 206 and codes[\"tail\"] == 206\n", + "\n", + "def fetch(url, filename=None):\n", + " \"\"\"Download a direct test-data link. Zips need unzipping afterwards.\"\"\"\n", + " dest = DATA / (filename or url.split(\"/\")[-1].split(\"?\")[0])\n", + " if not dest.exists():\n", + " with requests.get(url, headers=auth(), stream=True, timeout=600) as r:\n", + " r.raise_for_status()\n", + " with open(dest, \"wb\") as f:\n", + " shutil.copyfileobj(r.raw, f)\n", + " print(f\"{dest.name}: {dest.stat().st_size/1e6:.0f} MB\")\n", + " return dest\n", + "\n", + "def auth():\n", + " return {\"Authorization\": f\"Bearer {token}\"} # str(token) auto-refreshes\n", + " \n", + "def remote_size(url):\n", + " \"\"\"File size via a 1-byte ranged GET, since the endpoint may not do HEAD.\"\"\"\n", + " r = requests.get(url, headers={**auth(), \"Range\": \"bytes=0-0\"},\n", + " stream=True, timeout=60)\n", + " r.close()\n", + " if r.status_code == 206:\n", + " return int(r.headers[\"Content-Range\"].split(\"/\")[-1])\n", + " return int(r.headers[\"Content-Length\"])\n", + "\n", + "def open_remote(url):\n", + " \"\"\"Lazy-open a remote netCDF-4 over HTTP. Downloads nothing.\"\"\"\n", + " fs = fsspec.filesystem(\"https\", client_kwargs={\"headers\": auth()},\n", + " encoded=True, skip_instance_cache=True)\n", + " return xr.open_datatree(fs.open(url, block_size=4 * 2**20, size=remote_size(url)),\n", + " engine=\"h5netcdf\", phony_dims=\"access\",\n", + " decode_times=False)\n", + "\n", + "def download(prod, filename):\n", + " \"\"\"Fetch one file from the product through the entry endpoint.\"\"\"\n", + " dest = DATA / filename\n", + " if not dest.exists():\n", + " with prod.open(entry=filename) as src, open(dest, \"wb\") as dst:\n", + " shutil.copyfileobj(src, dst)\n", + " print(f\"{dest.name}: {dest.stat().st_size/1e6:.0f} MB\")\n", + " return dest\n", + "\n", + "def layout(path):\n", + " \"\"\"Chunk shape, chunk size, and codec for every variable in the file.\"\"\"\n", + " tree = xr.open_datatree(path, engine=\"h5netcdf\", phony_dims=\"access\", decode_times=False)\n", + " rows = []\n", + " for node in tree.subtree:\n", + " for name, v in node.ds.data_vars.items():\n", + " ch = v.encoding.get(\"chunksizes\")\n", + " rows.append({\"variable\": f\"{node.path}/{name}\", \"shape\": tuple(v.shape),\n", + " \"MB\": round(v.nbytes / 1e6, 2),\n", + " \"chunks\": tuple(ch) if ch else None,\n", + " \"chunk_MB\": round(v.dtype.itemsize * int(np.prod(ch)) / 1e6, 3) if ch else None,\n", + " \"codec\": v.encoding.get(\"compression\"),\n", + " \"shuffle\": bool(v.encoding.get(\"shuffle\"))})\n", + " return pd.DataFrame(rows).sort_values(\"MB\", ascending=False)" + ] + }, + { + "cell_type": "markdown", + "id": "9646e466", + "metadata": {}, + "source": [ + "## METimage (VII) level 1B radiances\n", + "\n", + "The full walkthrough. Each later instrument repeats these same five steps.\n", + "\n", + "First, the newest granule and its package contents. Expect one `.nc` and two\n", + "XML sidecars: the zip exists to carry those sidecars." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "184c83eb", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "W_XX-EUMETSAT-Darmstadt,SAT,SGA1-VII-1B-RAD_C_EUMT_20260916205124_G_O_20260916200259_20260916200400_C_N_T__ \n", + " files: ['W_XX-EUMETSAT-Darmstadt,SAT,SGA1-VII-1B-RAD_C_EUMT_20260916205124_G_O_20260916200259_20260916200400_C_N_T__.nc', 'EOPMetadata.xml', 'manifest.xml']\n" + ] + } + ], + "source": [ + "spec = PRODUCTS[\"METimage (VII)\"]\n", + "prod, entries = latest(spec[\"collection\"])\n", + "nc = next(e for e in entries if e.endswith(\".nc\"))\n", + "url = entry_url(prod, nc)" + ] + }, + { + "cell_type": "markdown", + "id": "5c5ca2b7", + "metadata": {}, + "source": [ + "Check range support on the per-file URL. Without 206 responses here, nothing\n", + "else in this notebook is possible and the product is download-only." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "11775c38", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "first bytes: 894844460d0a1a0a (894844... means HDF5/netCDF-4)\n", + "range status: {'head': 206, 'tail': 206}\n" + ] + }, + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 17, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "ranges_ok(url)" + ] + }, + { + "cell_type": "markdown", + "id": "c2cabe81", + "metadata": {}, + "source": [ + "Open the granule over HTTP. This reads metadata by range request and defers\n", + "everything else, so it should finish in seconds even though the file is over\n", + "100 MB. Then pull a 64x64 slice of one radiance channel to confirm that\n", + "subsetting works and to feel the per-read latency." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "6da1635d", + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 1.21 s, sys: 152 ms, total: 1.36 s\n", + "Wall time: 53.7 s\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
<xarray.DataTree>\n",
+       "Group: /\n",
+       "โ”‚   Attributes: (12/21)\n",
+       "โ”‚       title:                   VII L1B Radiances\n",
+       "โ”‚       Conventions:             CF-1.6\n",
+       "โ”‚       metadata_conventions:    Unidata Dataset Discovery v1.0\n",
+       "โ”‚       product_name:            W_XX-EUMETSAT-Darmstadt,SAT,SGA1-VII-1B-RAD_C_EU...\n",
+       "โ”‚       summary:                 VII/METimage L1B top of the atmosphere radiances\n",
+       "โ”‚       doi:                     \n",
+       "โ”‚       ...                      ...\n",
+       "โ”‚       sensing_start_time_utc:  2026-09-16 20:02:59.643\n",
+       "โ”‚       sensing_end_time_utc:    2026-09-16 20:04:00.134\n",
+       "โ”‚       environment:             Operational\n",
+       "โ”‚       references:              www.eumetsat.int\n",
+       "โ”‚       orbit_start:             5681\n",
+       "โ”‚       orbit_end:               5681\n",
+       "โ”œโ”€โ”€ Group: /status\n",
+       "โ”‚   โ”œโ”€โ”€ Group: /status/satellite\n",
+       "โ”‚   โ”‚       Dimensions:                   ()\n",
+       "โ”‚   โ”‚       Data variables: (12/24)\n",
+       "โ”‚   โ”‚           epoch_time_utc            datetime64[ns] 8B ...\n",
+       "โ”‚   โ”‚           semi_major_axis           float64 8B ...\n",
+       "โ”‚   โ”‚           eccentricity              float64 8B ...\n",
+       "โ”‚   โ”‚           inclination               float64 8B ...\n",
+       "โ”‚   โ”‚           perigee_argument          float64 8B ...\n",
+       "โ”‚   โ”‚           right_ascension           float64 8B ...\n",
+       "โ”‚   โ”‚           ...                        ...\n",
+       "โ”‚   โ”‚           z_velocity                float64 8B ...\n",
+       "โ”‚   โ”‚           yaw_error                 float64 8B ...\n",
+       "โ”‚   โ”‚           roll_error                float64 8B ...\n",
+       "โ”‚   โ”‚           pitch_error               float64 8B ...\n",
+       "โ”‚   โ”‚           leap_second_time_utc      datetime64[ns] 8B ...\n",
+       "โ”‚   โ”‚           leap_second_value         float32 4B ...\n",
+       "โ”‚   โ”œโ”€โ”€ Group: /status/instrument\n",
+       "โ”‚   โ”‚       Dimensions:              (mode_items: 1)\n",
+       "โ”‚   โ”‚       Dimensions without coordinates: mode_items\n",
+       "โ”‚   โ”‚       Data variables:\n",
+       "โ”‚   โ”‚           mode_start_time_utc  (mode_items) datetime64[ns] 8B ...\n",
+       "โ”‚   โ”‚           mode_end_time_utc    (mode_items) datetime64[ns] 8B ...\n",
+       "โ”‚   โ”‚           instrument_mode      (mode_items) <U4 16B ...\n",
+       "โ”‚   โ””โ”€โ”€ Group: /status/processing\n",
+       "โ”‚           Dimensions:            ()\n",
+       "โ”‚           Data variables:\n",
+       "โ”‚               creation_time_utc  datetime64[ns] 8B ...\n",
+       "โ”‚           Attributes:\n",
+       "โ”‚               processor_name:              VII_L1B\n",
+       "โ”‚               processor_version:           1.0\n",
+       "โ”‚               processing_mode:             NRT\n",
+       "โ”‚               format_version:              6.0\n",
+       "โ”‚               auxiliary_data_version:      EUM/LEO-EPSSG/SPE/14/777147 v4A\n",
+       "โ”‚               pgs_reference_and_version:   EUM/LEO-EPSSG/DOC/14/746628 v5\n",
+       "โ”‚               pfs_reference_and_version:   EUM/LEO-EPSSG/SPE/14/777138 v5\n",
+       "โ”‚               atbd_reference_and_version:  EUM/LEO-EPSSG/DOC/13/702485 v4\n",
+       "โ”‚               source:                      ['SGA1_VII_1B_AUX_LMDB___S20250813000000Z_Ex...\n",
+       "โ”œโ”€โ”€ Group: /data\n",
+       "โ”‚   โ”œโ”€โ”€ Group: /data/measurement_data\n",
+       "โ”‚   โ”‚       Dimensions:              (num_tie_points_alt: 140, num_tie_points_act: 394,\n",
+       "โ”‚   โ”‚                                 num_lines: 840, num_pixels: 3144, num_scans: 35)\n",
+       "โ”‚   โ”‚       Dimensions without coordinates: num_tie_points_alt, num_tie_points_act,\n",
+       "โ”‚   โ”‚                                       num_lines, num_pixels, num_scans\n",
+       "โ”‚   โ”‚       Data variables: (12/32)\n",
+       "โ”‚   โ”‚           latitude             (num_tie_points_alt, num_tie_points_act) float64 441kB ...\n",
+       "โ”‚   โ”‚           longitude            (num_tie_points_alt, num_tie_points_act) float64 441kB ...\n",
+       "โ”‚   โ”‚           delta_lat_N_dem      (num_lines, num_pixels) float32 11MB ...\n",
+       "โ”‚   โ”‚           delta_lon_E_dem      (num_lines, num_pixels) float32 11MB ...\n",
+       "โ”‚   โ”‚           solar_zenith         (num_tie_points_alt, num_tie_points_act) float64 441kB ...\n",
+       "โ”‚   โ”‚           solar_azimuth        (num_tie_points_alt, num_tie_points_act) float64 441kB ...\n",
+       "โ”‚   โ”‚           ...                   ...\n",
+       "โ”‚   โ”‚           vii_6725             (num_lines, num_pixels) float32 11MB ...\n",
+       "โ”‚   โ”‚           vii_7325             (num_lines, num_pixels) float32 11MB ...\n",
+       "โ”‚   โ”‚           vii_8540             (num_lines, num_pixels) float32 11MB ...\n",
+       "โ”‚   โ”‚           vii_10690            (num_lines, num_pixels) float32 11MB ...\n",
+       "โ”‚   โ”‚           vii_12020            (num_lines, num_pixels) float32 11MB ...\n",
+       "โ”‚   โ”‚           vii_13345            (num_lines, num_pixels) float32 11MB ...\n",
+       "โ”‚   โ”œโ”€โ”€ Group: /data/calibration_data\n",
+       "โ”‚   โ”‚       Dimensions:                         (isrf_samples_443: 121,\n",
+       "โ”‚   โ”‚                                            isrf_samples_555: 81,\n",
+       "โ”‚   โ”‚                                            isrf_samples_668: 81,\n",
+       "โ”‚   โ”‚                                            isrf_samples_752: 71,\n",
+       "โ”‚   โ”‚                                            isrf_samples_763: 71,\n",
+       "โ”‚   โ”‚                                            isrf_samples_865: 121,\n",
+       "โ”‚   โ”‚                                            ...\n",
+       "โ”‚   โ”‚                                            isrf_samples_7325: 160,\n",
+       "โ”‚   โ”‚                                            isrf_samples_8540: 125,\n",
+       "โ”‚   โ”‚                                            isrf_samples_10690: 124,\n",
+       "โ”‚   โ”‚                                            isrf_samples_12020: 69,\n",
+       "โ”‚   โ”‚                                            isrf_samples_13345: 51,\n",
+       "โ”‚   โ”‚                                            num_chan_solar: 11, num_chan_thermal: 9)\n",
+       "โ”‚   โ”‚       Dimensions without coordinates: isrf_samples_443, isrf_samples_555,\n",
+       "โ”‚   โ”‚                                       isrf_samples_668, isrf_samples_752,\n",
+       "โ”‚   โ”‚                                       isrf_samples_763, isrf_samples_865,\n",
+       "โ”‚   โ”‚                                       isrf_samples_914, isrf_samples_1240,\n",
+       "โ”‚   โ”‚                                       isrf_samples_1375, isrf_samples_1630,\n",
+       "โ”‚   โ”‚                                       isrf_samples_2250, isrf_samples_3740,\n",
+       "โ”‚   โ”‚                                       isrf_samples_3959, isrf_samples_4050,\n",
+       "โ”‚   โ”‚                                       isrf_samples_6725, isrf_samples_7325,\n",
+       "โ”‚   โ”‚                                       isrf_samples_8540, isrf_samples_10690,\n",
+       "โ”‚   โ”‚                                       isrf_samples_12020, isrf_samples_13345,\n",
+       "โ”‚   โ”‚                                       num_chan_solar, num_chan_thermal\n",
+       "โ”‚   โ”‚       Data variables: (12/44)\n",
+       "โ”‚   โ”‚           ISRF_443                        (isrf_samples_443) float32 484B ...\n",
+       "โ”‚   โ”‚           ISRF_443_wavelength             (isrf_samples_443) float64 968B ...\n",
+       "โ”‚   โ”‚           ISRF_555                        (isrf_samples_555) float32 324B ...\n",
+       "โ”‚   โ”‚           ISRF_555_wavelength             (isrf_samples_555) float64 648B ...\n",
+       "โ”‚   โ”‚           ISRF_668                        (isrf_samples_668) float32 324B ...\n",
+       "โ”‚   โ”‚           ISRF_668_wavelength             (isrf_samples_668) float64 648B ...\n",
+       "โ”‚   โ”‚           ...                              ...\n",
+       "โ”‚   โ”‚           ISRF_13345                      (isrf_samples_13345) float32 204B ...\n",
+       "โ”‚   โ”‚           ISRF_13345_wavelength           (isrf_samples_13345) float64 408B ...\n",
+       "โ”‚   โ”‚           band_averaged_solar_irradiance  (num_chan_solar) float32 44B ...\n",
+       "โ”‚   โ”‚           channel_cw_thermal              (num_chan_thermal) float32 36B ...\n",
+       "โ”‚   โ”‚           bt_conversion_a                 (num_chan_thermal) float32 36B ...\n",
+       "โ”‚   โ”‚           bt_conversion_b                 (num_chan_thermal) float32 36B ...\n",
+       "โ”‚   โ”œโ”€โ”€ Group: /data/quality_information\n",
+       "โ”‚   โ”‚       Dimensions:              (num_scans: 35, num_chan: 20)\n",
+       "โ”‚   โ”‚       Dimensions without coordinates: num_scans, num_chan\n",
+       "โ”‚   โ”‚       Data variables:\n",
+       "โ”‚   โ”‚           data_quality         (num_scans, num_chan) uint8 700B ...\n",
+       "โ”‚   โ”‚           calibration_quality  (num_scans, num_chan) uint16 1kB ...\n",
+       "โ”‚   โ”‚           geolocation_quality  (num_scans) uint8 35B ...\n",
+       "โ”‚   โ””โ”€โ”€ Group: /data/processing_flags\n",
+       "โ”‚           Dimensions:                 (num_scans: 35, num_chan: 20, num_pixels_alt: 24,\n",
+       "โ”‚                                        num_pixels: 3144)\n",
+       "โ”‚           Dimensions without coordinates: num_scans, num_chan, num_pixels_alt, num_pixels\n",
+       "โ”‚           Data variables:\n",
+       "โ”‚               vii_processing_flag     (num_scans, num_chan) uint8 700B ...\n",
+       "โ”‚               pixel_duplication_mask  (num_pixels_alt, num_pixels) uint8 75kB ...\n",
+       "โ”‚               pixel_performance_mask  (num_chan, num_pixels_alt) uint8 480B ...\n",
+       "โ”‚               processing_mode         uint8 1B ...\n",
+       "โ””โ”€โ”€ Group: /quality\n",
+       "        Dimensions:                    ()\n",
+       "        Data variables:\n",
+       "            duration_of_product        float64 8B ...\n",
+       "            duration_of_data_present   float64 8B ...\n",
+       "            duration_of_data_missing   float64 8B ...\n",
+       "            duration_of_data_degraded  float64 8B ...\n",
+       "        Attributes:\n",
+       "            overall_quality_flag:  0
" + ], + "text/plain": [ + "\n", + "Group: /\n", + "โ”‚ Attributes: (12/21)\n", + "โ”‚ title: VII L1B Radiances\n", + "โ”‚ Conventions: CF-1.6\n", + "โ”‚ metadata_conventions: Unidata Dataset Discovery v1.0\n", + "โ”‚ product_name: W_XX-EUMETSAT-Darmstadt,SAT,SGA1-VII-1B-RAD_C_EU...\n", + "โ”‚ summary: VII/METimage L1B top of the atmosphere radiances\n", + "โ”‚ doi: \n", + "โ”‚ ... ...\n", + "โ”‚ sensing_start_time_utc: 2026-09-16 20:02:59.643\n", + "โ”‚ sensing_end_time_utc: 2026-09-16 20:04:00.134\n", + "โ”‚ environment: Operational\n", + "โ”‚ references: www.eumetsat.int\n", + "โ”‚ orbit_start: 5681\n", + "โ”‚ orbit_end: 5681\n", + "โ”œโ”€โ”€ Group: /status\n", + "โ”‚ โ”œโ”€โ”€ Group: /status/satellite\n", + "โ”‚ โ”‚ Dimensions: ()\n", + "โ”‚ โ”‚ Data variables: (12/24)\n", + "โ”‚ โ”‚ epoch_time_utc datetime64[ns] 8B ...\n", + "โ”‚ โ”‚ semi_major_axis float64 8B ...\n", + "โ”‚ โ”‚ eccentricity float64 8B ...\n", + "โ”‚ โ”‚ inclination float64 8B ...\n", + "โ”‚ โ”‚ perigee_argument float64 8B ...\n", + "โ”‚ โ”‚ right_ascension float64 8B ...\n", + "โ”‚ โ”‚ ... ...\n", + "โ”‚ โ”‚ z_velocity float64 8B ...\n", + "โ”‚ โ”‚ yaw_error float64 8B ...\n", + "โ”‚ โ”‚ roll_error float64 8B ...\n", + "โ”‚ โ”‚ pitch_error float64 8B ...\n", + "โ”‚ โ”‚ leap_second_time_utc datetime64[ns] 8B ...\n", + "โ”‚ โ”‚ leap_second_value float32 4B ...\n", + "โ”‚ โ”œโ”€โ”€ Group: /status/instrument\n", + "โ”‚ โ”‚ Dimensions: (mode_items: 1)\n", + "โ”‚ โ”‚ Dimensions without coordinates: mode_items\n", + "โ”‚ โ”‚ Data variables:\n", + "โ”‚ โ”‚ mode_start_time_utc (mode_items) datetime64[ns] 8B ...\n", + "โ”‚ โ”‚ mode_end_time_utc (mode_items) datetime64[ns] 8B ...\n", + "โ”‚ โ”‚ instrument_mode (mode_items) \n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
variableshapeMBchunkschunk_MBcodecshuffle
46/data/measurement_data/vii_668(840, 3144)10.56(1, 3144)0.013NoneFalse
45/data/measurement_data/vii_555(840, 3144)10.56(1, 3144)0.013NoneFalse
34/data/measurement_data/delta_lat_N_dem(840, 3144)10.56(1, 3144)0.013NoneFalse
35/data/measurement_data/delta_lon_E_dem(840, 3144)10.56(1, 3144)0.013NoneFalse
50/data/measurement_data/vii_914(840, 3144)10.56(1, 3144)0.013NoneFalse
48/data/measurement_data/vii_763(840, 3144)10.56(1, 3144)0.013NoneFalse
49/data/measurement_data/vii_865(840, 3144)10.56(1, 3144)0.013NoneFalse
47/data/measurement_data/vii_752(840, 3144)10.56(1, 3144)0.013NoneFalse
44/data/measurement_data/vii_443(840, 3144)10.56(1, 3144)0.013NoneFalse
60/data/measurement_data/vii_8540(840, 3144)10.56(1, 3144)0.013NoneFalse
59/data/measurement_data/vii_7325(840, 3144)10.56(1, 3144)0.013NoneFalse
58/data/measurement_data/vii_6725(840, 3144)10.56(1, 3144)0.013NoneFalse
57/data/measurement_data/vii_4050(840, 3144)10.56(1, 3144)0.013NoneFalse
56/data/measurement_data/vii_3959(840, 3144)10.56(1, 3144)0.013NoneFalse
55/data/measurement_data/vii_3740(840, 3144)10.56(1, 3144)0.013NoneFalse
\n", + "" + ], + "text/plain": [ + " variable shape MB chunks \\\n", + "46 /data/measurement_data/vii_668 (840, 3144) 10.56 (1, 3144) \n", + "45 /data/measurement_data/vii_555 (840, 3144) 10.56 (1, 3144) \n", + "34 /data/measurement_data/delta_lat_N_dem (840, 3144) 10.56 (1, 3144) \n", + "35 /data/measurement_data/delta_lon_E_dem (840, 3144) 10.56 (1, 3144) \n", + "50 /data/measurement_data/vii_914 (840, 3144) 10.56 (1, 3144) \n", + "48 /data/measurement_data/vii_763 (840, 3144) 10.56 (1, 3144) \n", + "49 /data/measurement_data/vii_865 (840, 3144) 10.56 (1, 3144) \n", + "47 /data/measurement_data/vii_752 (840, 3144) 10.56 (1, 3144) \n", + "44 /data/measurement_data/vii_443 (840, 3144) 10.56 (1, 3144) \n", + "60 /data/measurement_data/vii_8540 (840, 3144) 10.56 (1, 3144) \n", + "59 /data/measurement_data/vii_7325 (840, 3144) 10.56 (1, 3144) \n", + "58 /data/measurement_data/vii_6725 (840, 3144) 10.56 (1, 3144) \n", + "57 /data/measurement_data/vii_4050 (840, 3144) 10.56 (1, 3144) \n", + "56 /data/measurement_data/vii_3959 (840, 3144) 10.56 (1, 3144) \n", + "55 /data/measurement_data/vii_3740 (840, 3144) 10.56 (1, 3144) \n", + "\n", + " chunk_MB codec shuffle \n", + "46 0.013 None False \n", + "45 0.013 None False \n", + "34 0.013 None False \n", + "35 0.013 None False \n", + "50 0.013 None False \n", + "48 0.013 None False \n", + "49 0.013 None False \n", + "47 0.013 None False \n", + "44 0.013 None False \n", + "60 0.013 None False \n", + "59 0.013 None False \n", + "58 0.013 None False \n", + "57 0.013 None False \n", + "56 0.013 None False \n", + "55 0.013 None False " + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "local = download(prod, nc)\n", + "L = layout(local)\n", + "L.head(15)" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "92435aab", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "compressed variables: 0 of 115\n", + "logical size: 235 MB on disk: 119 MB\n", + "chunk sizes (MB): {'min': 0.0, '50%': 0.013, 'max': 0.013}\n" + ] + } + ], + "source": [ + "print(\"compressed variables:\", int(L.codec.notna().sum()), \"of\", len(L))\n", + "print(\"logical size:\", round(L.MB.sum()), \"MB on disk:\", round(local.stat().st_size/1e6), \"MB\")\n", + "print(\"chunk sizes (MB):\", L.chunk_MB.describe()[[\"min\", \"50%\", \"max\"]].round(3).to_dict())" + ] + }, + { + "cell_type": "markdown", + "id": "df387104", + "metadata": {}, + "source": [ + "Measure what internal compression would buy. This compresses one real chunk\n", + "on local CPU. On float radiances, deflate level 1 with byte shuffle is the\n", + "cheap, universally readable option; zstd does the same job faster. Ratios\n", + "above about 1.5x make a solid case." + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "7697e069", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "/data/measurement_data/vii_668 (512, 512) float32\n", + "deflate-1: 1.93x, with shuffle 1.35x\n", + "zstd-3: 1.59x, with shuffle 1.35x\n" + ] + } + ], + "source": [ + "import zstandard as zstd\n", + "row = L.dropna(subset=[\"chunks\"]).iloc[0]\n", + "arr = np.ascontiguousarray(\n", + " tree[str(Path(row.variable).parent)].ds[Path(row.variable).name][:512, :512].values)\n", + "raw, shuf = arr.tobytes(), arr.view(np.uint8).reshape(-1, arr.dtype.itemsize).T.tobytes()\n", + "\n", + "print(row.variable, arr.shape, arr.dtype)\n", + "print(f\"deflate-1: {len(raw)/len(zlib.compress(raw,1)):.2f}x, \"\n", + " f\"with shuffle {len(raw)/len(zlib.compress(shuf,1)):.2f}x\")\n", + "c = zstd.ZstdCompressor(level=3)\n", + "print(f\"zstd-3: {len(raw)/len(c.compress(raw)):.2f}x, \"\n", + " f\"with shuffle {len(raw)/len(c.compress(shuf)):.2f}x\")" + ] + }, + { + "cell_type": "markdown", + "id": "9ded0d0e", + "metadata": {}, + "source": [ + "## MWS\n", + "\n", + "Microwave sounder, 24 channels, small granules. The same steps usually run in\n", + "under a minute end to end." + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "39b5e427", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "W_XX-EUMETSAT-Darmstadt,SAT,SGA1-MWS-1B-RAD_C_EUMT_20260916212055_G_O_20260916203857_20260916204158_O_N____ \n", + " files: ['W_XX-EUMETSAT-Darmstadt,SAT,SGA1-MWS-1B-RAD_C_EUMT_20260916212055_G_O_20260916203857_20260916204158_O_N____.nc', 'EOPMetadata.xml', 'manifest.xml']\n", + "first bytes: 894844460d0a1a0a (894844... means HDF5/netCDF-4)\n", + "range status: {'head': 206, 'tail': 206}\n" + ] + }, + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 29, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "spec = PRODUCTS[\"MWS\"]\n", + "if spec.get(\"collection\"):\n", + " prod, entries = latest(spec[\"collection\"])\n", + " datafile = next(e for e in entries if not e.lower().endswith((\".xml\", \".txt\")))\n", + " url = entry_url(prod, datafile)\n", + "else:\n", + " url = spec[\"url\"] # direct test-data link\n", + " assert url, \"paste this product's link from the test-data page into PRODUCTS\"\n", + "\n", + "ranges_ok(url)" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "6041c750", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "W_XX-EUMETSAT-Darmstadt,SAT,SGA1-MWS-1B-RAD_C_EUMT_20260916212055_G_O_20260916203857_20260916204158_O_N____.nc: 3 MB\n", + "compressed: 0 of 167 median chunk MB: 0.1075\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
variableshapeMBchunkschunk_MBcodecshuffle
130/data/calibration/mws_toa_radiance(80, 95, 24)1.46(28, 95, 24)0.511NoneFalse
108/data/calibration/mws_toa_brightness_temperature(80, 95, 24)1.46(28, 95, 24)0.511NoneFalse
136/data/measurement/mws_earth_view_counts(80, 95, 24)0.73(57, 95, 24)0.520NoneFalse
140/data/processing_information/mws_radiance_flag(80, 95, 24)0.18(80, 95, 24)0.182NoneFalse
162/data/processing_information/mws_brightnesstem...(80, 95, 24)0.18(80, 95, 24)0.182NoneFalse
102/data/navigation/mws_surface_type(80, 95, 2)0.12NoneNaNNoneFalse
99/data/navigation/mws_solar_azimuth_angle(80, 95)0.06NoneNaNNoneFalse
98/data/navigation/mws_satellite_zenith_angle(80, 95)0.06NoneNaNNoneFalse
104/data/navigation/mws_lat(80, 95)0.06NoneNaNNoneFalse
135/data/measurement/mws_earth_view_counts_os_stdev(80, 95, 2)0.06NoneNaNNoneFalse
\n", + "
" + ], + "text/plain": [ + " variable shape MB \\\n", + "130 /data/calibration/mws_toa_radiance (80, 95, 24) 1.46 \n", + "108 /data/calibration/mws_toa_brightness_temperature (80, 95, 24) 1.46 \n", + "136 /data/measurement/mws_earth_view_counts (80, 95, 24) 0.73 \n", + "140 /data/processing_information/mws_radiance_flag (80, 95, 24) 0.18 \n", + "162 /data/processing_information/mws_brightnesstem... (80, 95, 24) 0.18 \n", + "102 /data/navigation/mws_surface_type (80, 95, 2) 0.12 \n", + "99 /data/navigation/mws_solar_azimuth_angle (80, 95) 0.06 \n", + "98 /data/navigation/mws_satellite_zenith_angle (80, 95) 0.06 \n", + "104 /data/navigation/mws_lat (80, 95) 0.06 \n", + "135 /data/measurement/mws_earth_view_counts_os_stdev (80, 95, 2) 0.06 \n", + "\n", + " chunks chunk_MB codec shuffle \n", + "130 (28, 95, 24) 0.511 None False \n", + "108 (28, 95, 24) 0.511 None False \n", + "136 (57, 95, 24) 0.520 None False \n", + "140 (80, 95, 24) 0.182 None False \n", + "162 (80, 95, 24) 0.182 None False \n", + "102 None NaN None False \n", + "99 None NaN None False \n", + "98 None NaN None False \n", + "104 None NaN None False \n", + "135 None NaN None False " + ] + }, + "execution_count": 30, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "tree = open_remote(url)\n", + "local = download(prod, datafile) if spec.get(\"collection\") else fetch(url)\n", + "L = layout(local)\n", + "print(\"compressed:\", int(L.codec.notna().sum()), \"of\", len(L),\n", + " \" median chunk MB:\", L.chunk_MB.median())\n", + "L.head(10)" + ] + }, + { + "cell_type": "markdown", + "id": "ad377891", + "metadata": {}, + "source": [ + "## Radio Occultation\n", + "\n", + "GRAS-2 has a Data Store collection, and its title says netCDF. The\n", + "`ranges_ok` call prints the first bytes as a check anyway: `894844` opens\n", + "like the others, `425546` spells BUFR, which has no internal chunking to\n", + "assess and would end this section early as its own finding.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "b36ca262", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "W_XX-EUMETSAT-Darmstadt,SAT,SGA1-RO_-1B-BND_C_EUMT_20260916211844_G_O_20260916202243_20260916202900_O_N_G20 \n", + " files: ['W_XX-EUMETSAT-Darmstadt,SAT,SGA1-RO_-1B-BND_C_EUMT_20260916211844_G_O_20260916202243_20260916202900_O_N_G20.nc', 'EOPMetadata.xml', 'manifest.xml']\n", + "first bytes: 894844460d0a1a0a (894844... means HDF5/netCDF-4)\n", + "range status: {'head': 206, 'tail': 206}\n" + ] + }, + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 32, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "spec = PRODUCTS[\"Radio Occultation\"]\n", + "if spec.get(\"collection\"):\n", + " prod, entries = latest(spec[\"collection\"])\n", + " datafile = next(e for e in entries if not e.lower().endswith((\".xml\", \".txt\")))\n", + " url = entry_url(prod, datafile)\n", + "else:\n", + " url = spec[\"url\"] # direct test-data link\n", + " assert url, \"paste this product's link from the test-data page into PRODUCTS\"\n", + "\n", + "ranges_ok(url)" + ] + }, + { + "cell_type": "markdown", + "id": "5269ed83-aff3-4002-8a95-2dcde92392e5", + "metadata": {}, + "source": [ + "### Read Efficiency\n", + "This section tests the read efficiency for hosted vs local read for the \"EO:EUM:DAT:0464\" product." + ] + }, + { + "cell_type": "code", + "execution_count": 56, + "id": "a2a610ee-4730-4e4f-a846-620362dcc977", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "variable : vii_443 (840, 3144) float32\n", + "chunks : (1, 3144) codec=None\n", + "file : 119 MB\n", + "\n", + "local slice : 16 KB in 8 ms\n", + "\n", + "open : 75.5 MB in 12 reads, 21.7s\n", + "slice : 16.7 MB in 2 reads, 6.5s -> 1019x amplification for 16 KB wanted\n", + "total : 92.2 MB of a 119 MB file\n" + ] + } + ], + "source": [ + "import time, urllib.parse\n", + "from pathlib import Path\n", + "import xarray as xr, fsspec, requests\n", + "\n", + "LOCAL = Path(\"data/W_XX-EUMETSAT-Darmstadt,SAT,SGA1-VII-1B-RAD_C_EUMT_20260916205124_G_O_20260916200259_20260916200400_C_N_T__.nc\")\n", + "enc = lambda s: urllib.parse.quote(s, safe=\"\")\n", + "\n", + "pid = LOCAL.name[:-3] # product id = filename minus .nc\n", + "base = (\"https://api.eumetsat.int/data/download/1.0.0/collections/\"\n", + " f\"{enc('EO:EUM:DAT:0464')}/products/{enc(pid)}\")\n", + "URL = f\"{base}/entry?name={enc(LOCAL.name)}\"\n", + "\n", + "def remote_size(u):\n", + " r = requests.get(u, headers={**auth(), \"Range\": \"bytes=0-0\"}, stream=True, timeout=60); r.close()\n", + " return int(r.headers[\"Content-Range\"].split(\"/\")[-1]) if r.status_code == 206 \\\n", + " else int(r.headers[\"Content-Length\"])\n", + "\n", + "# ---- pick the target variable from the local copy -------------------------\n", + "lt = xr.open_datatree(LOCAL, engine=\"h5netcdf\", phony_dims=\"access\", decode_times=False)\n", + "rad = lt[\"data/measurement_data\"].ds\n", + "var = \"vii_443\"\n", + "ny, nx = rad[var].shape[:2]\n", + "sl = (slice(ny//2, ny//2+64), slice(nx//2, nx//2+64)) # mid-swath, not a corner\n", + "chunks = rad[var].encoding.get(\"chunksizes\")\n", + "print(f\"variable : {var} {rad[var].shape} {rad[var].dtype}\")\n", + "print(f\"chunks : {chunks} codec={rad[var].encoding.get('compression')}\")\n", + "print(f\"file : {LOCAL.stat().st_size/1e6:.0f} MB\\n\")\n", + "\n", + "# ---- local baseline -------------------------------------------------------\n", + "t0 = time.perf_counter(); a = rad[var][sl].values; t_local = time.perf_counter()-t0\n", + "wanted = a.nbytes\n", + "print(f\"local slice : {wanted/1e3:.0f} KB in {t_local*1e3:.0f} ms\\n\")\n", + "\n", + "bs = 8*1024*1024\n", + "fs = fsspec.filesystem(\"https\", client_kwargs={\"headers\": auth()},\n", + " encoded=True, skip_instance_cache=True)\n", + "f = fs.open(URL, block_size=bs, size=remote_size(URL))\n", + "\n", + "t0 = time.perf_counter()\n", + "rt = xr.open_datatree(f, engine=\"h5netcdf\", phony_dims=\"access\", decode_times=False)\n", + "t_open = time.perf_counter() - t0\n", + "open_bytes, open_reqs = f.cache.total_requested_bytes, f.cache.miss_count\n", + "\n", + "t0 = time.perf_counter()\n", + "b = rt[\"data/measurement_data\"].ds[var][sl].values\n", + "t_read = time.perf_counter() - t0\n", + "read_bytes = f.cache.total_requested_bytes - open_bytes\n", + "read_reqs = f.cache.miss_count - open_reqs\n", + "\n", + "print(f\"open : {open_bytes/1e6:6.1f} MB in {open_reqs:3d} reads, {t_open:5.1f}s\")\n", + "print(f\"slice : {read_bytes/1e6:6.1f} MB in {read_reqs:3d} reads, {t_read:5.1f}s\"\n", + " f\" -> {read_bytes/wanted:.0f}x amplification for {wanted/1e3:.0f} KB wanted\")\n", + "print(f\"total : {(open_bytes+read_bytes)/1e6:6.1f} MB of a \"\n", + " f\"{LOCAL.stat().st_size/1e6:.0f} MB file\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "89f82bc0-87a6-4637-9973-e90a83ce8db0", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python [conda env:notebook] *", + "language": "python", + "name": "conda-env-notebook-py" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} From 8603fa97377cefe0c7908e0cd6bb4657bd8dee34 Mon Sep 17 00:00:00 2001 From: omshinde Date: Tue, 22 Sep 2026 16:17:08 +0000 Subject: [PATCH 2/5] Add virtual zarr testing --- eumetsat_metopsg/metopsg_cloud_native.ipynb | 142 +++++++++++++++++++- 1 file changed, 140 insertions(+), 2 deletions(-) diff --git a/eumetsat_metopsg/metopsg_cloud_native.ipynb b/eumetsat_metopsg/metopsg_cloud_native.ipynb index 43bafbc..222a2cd 100644 --- a/eumetsat_metopsg/metopsg_cloud_native.ipynb +++ b/eumetsat_metopsg/metopsg_cloud_native.ipynb @@ -62,7 +62,7 @@ }, { "cell_type": "code", - "execution_count": 44, + "execution_count": 2, "id": "c733c202", "metadata": {}, "outputs": [ @@ -1905,11 +1905,149 @@ " f\"{LOCAL.stat().st_size/1e6:.0f} MB file\")" ] }, + { + "cell_type": "markdown", + "id": "f65f8430-3766-4f7f-8dde-316d106e2038", + "metadata": {}, + "source": [ + "### Making Virtual Zarr" + ] + }, { "cell_type": "code", - "execution_count": null, + "execution_count": 3, "id": "89f82bc0-87a6-4637-9973-e90a83ce8db0", "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "https://api.eumetsat.int/data/download/1.0.0/collections/EO%3AEUM%3ADAT%3A0464/products/W_XX-EUMETSAT-Darmstadt%2CSAT%2CSGA1-VII-1B-RAD_C_EUMT_20260916205124_G_O_20260916200259_20260916200400_C_N_T__/entry?name=W_XX-EUMETSAT-Darmstadt%2CSAT%2CSGA1-VII-1B-RAD_C_EUMT_20260916205124_G_O_20260916200259_20260916200400_C_N_T__.nc\n" + ] + } + ], + "source": [ + "import urllib.parse, eumdac\n", + "from pathlib import Path\n", + "\n", + "LOCAL = Path(\"data/W_XX-EUMETSAT-Darmstadt,SAT,SGA1-VII-1B-RAD_C_EUMT_20260916205124_G_O_20260916200259_20260916200400_C_N_T__.nc\")\n", + "enc = lambda s: urllib.parse.quote(s, safe=\"\")\n", + "\n", + "KEY, SECRET = Path.home().joinpath(\".eumdac\", \"credentials\").read_text().strip().split(\",\")\n", + "token = eumdac.AccessToken((KEY, SECRET))\n", + "\n", + "pid = LOCAL.name[:-3]\n", + "url = (f\"https://api.eumetsat.int/data/download/1.0.0/collections/{enc('EO:EUM:DAT:0464')}\"\n", + " f\"/products/{enc(pid)}/entry?name={enc(LOCAL.name)}\")\n", + "print(url)" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "f9abbfdc-047e-4c39-a665-d0465964ea76", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "True\n" + ] + } + ], + "source": [ + "import shutil\n", + "from pathlib import Path\n", + "\n", + "root = Path(\"data\").resolve()\n", + "src = root / \"W_XX-EUMETSAT-Darmstadt,SAT,SGA1-VII-1B-RAD_C_EUMT_20260916205124_G_O_20260916200259_20260916200400_C_N_T__.nc\"\n", + "LOCAL = root / \"metimage_l1b_sample.nc\"\n", + "if not LOCAL.exists():\n", + " shutil.copy2(src, LOCAL) # or src.link_to / os.link to avoid duplicating 119 MB\n", + "print(LOCAL.exists())" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "73929cc4-b88f-499e-afeb-7be304a5145b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " Size: 118MB\n", + "Dimensions: (num_tie_points_alt: 140, num_tie_points_act: 394,\n", + " data/num_lines: 840, data/num_pixels: 3144,\n", + " data/num_scans: 35)\n", + "Dimensions without coordinates: num_tie_points_alt, num_tie_points_act,\n", + " data/num_lines, data/num_pixels, data/num_scans\n", + "Data variables: (12/32)\n", + " latitude (num_tie_points_alt, num_tie_points_act) int32 221kB ManifestArray Date: Tue, 22 Sep 2026 12:04:10 -0500 Subject: [PATCH 3/5] addressed review fixed shuffle demo --- eumetsat_metopsg/metopsg_cloud_native.ipynb | 1567 +++++++++++++------ 1 file changed, 1079 insertions(+), 488 deletions(-) diff --git a/eumetsat_metopsg/metopsg_cloud_native.ipynb b/eumetsat_metopsg/metopsg_cloud_native.ipynb index 222a2cd..17d220c 100644 --- a/eumetsat_metopsg/metopsg_cloud_native.ipynb +++ b/eumetsat_metopsg/metopsg_cloud_native.ipynb @@ -25,7 +25,7 @@ "2. Check whether the hosting answers HTTP range requests\n", "3. Open a granule lazily over HTTP with xarray, without downloading it\n", "4. Download one granule and read its chunking and compression settings\n", - "5. Build a kerchunk index and read the remote file through the Zarr engine" + "5. Build a reference file with [VirtualiZarr](https://virtualizarr.readthedocs.io/) (and optionally write kerchunk JSON) so the remote granule can be read through the Zarr engine\n" ] }, { @@ -44,12 +44,11 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "id": "f1e68a81", "metadata": {}, "outputs": [ { - "name": "stdout", "output_type": "stream", "text": [ "Note: you may need to restart the kernel to use updated packages.\n" @@ -57,12 +56,12 @@ } ], "source": [ - "%pip install -q eumdac \"xarray>=2024.10\" h5netcdf fsspec aiohttp kerchunk zarr pandas requests zstandard" + "%pip install -q eumdac \"xarray>=2024.10\" h5netcdf fsspec aiohttp kerchunk zarr pandas requests zstandard virtualizarr obstore\n" ] }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 16, "id": "c733c202", "metadata": {}, "outputs": [ @@ -124,7 +123,7 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 17, "id": "021bc277", "metadata": {}, "outputs": [], @@ -145,20 +144,13 @@ "source": [ "## Helper functions\n", "\n", - "Five short functions used in every section. `latest` fetches the newest\n", - "product in a collection and lists the files inside its delivery package.\n", - "`ranges_ok` sends two ranged GETs; the suffix range matters because zip\n", - "central directories and Parquet footers sit at the end of the file, so a\n", - "server that cannot serve file tails cannot serve those formats in place.\n", - "`open_remote` opens a URL with xarray through fsspec, reading only the bytes\n", - "h5netcdf asks for. `layout` tabulates chunk shape and codec per variable.\n", - "`fetch` downloads a direct test-data link. `try_kerchunk` indexes the chunks of a local copy, points the references at\n", - "the remote URL, and reads one slice back through the Zarr engine." + "Short helpers used in every section. Prefer the function docstrings below over\n", + "re-describing them here.\n" ] }, { "cell_type": "code", - "execution_count": 47, + "execution_count": 18, "id": "999c672d", "metadata": {}, "outputs": [], @@ -199,9 +191,6 @@ " print(f\"{dest.name}: {dest.stat().st_size/1e6:.0f} MB\")\n", " return dest\n", "\n", - "def auth():\n", - " return {\"Authorization\": f\"Bearer {token}\"} # str(token) auto-refreshes\n", - " \n", "def remote_size(url):\n", " \"\"\"File size via a 1-byte ranged GET, since the endpoint may not do HEAD.\"\"\"\n", " r = requests.get(url, headers={**auth(), \"Range\": \"bytes=0-0\"},\n", @@ -211,13 +200,23 @@ " return int(r.headers[\"Content-Range\"].split(\"/\")[-1])\n", " return int(r.headers[\"Content-Length\"])\n", "\n", - "def open_remote(url):\n", - " \"\"\"Lazy-open a remote netCDF-4 over HTTP. Downloads nothing.\"\"\"\n", + "def open_remote(url, log_requests=True):\n", + " \"\"\"Lazy-open a remote netCDF-4 over HTTP. Downloads nothing.\n", + "\n", + " Uses fsspec blockcache so ``block_size`` actually batches range reads.\n", + " When ``log_requests`` is True, prints how many range requests and how many\n", + " bytes the open itself required.\n", + " \"\"\"\n", " fs = fsspec.filesystem(\"https\", client_kwargs={\"headers\": auth()},\n", " encoded=True, skip_instance_cache=True)\n", - " return xr.open_datatree(fs.open(url, block_size=4 * 2**20, size=remote_size(url)),\n", - " engine=\"h5netcdf\", phony_dims=\"access\",\n", + " f = fs.open(url, mode=\"rb\", block_size=4 * 2**20, cache_type=\"blockcache\",\n", + " size=remote_size(url))\n", + " tree = xr.open_datatree(f, engine=\"h5netcdf\", phony_dims=\"access\",\n", " decode_times=False)\n", + " if log_requests and getattr(f, \"cache\", None) is not None:\n", + " print(f\"open_remote: {f.cache.miss_count} range requests, \"\n", + " f\"{f.cache.total_requested_bytes / 1e6:.1f} MB transferred\")\n", + " return tree\n", "\n", "def download(prod, filename):\n", " \"\"\"Fetch one file from the product through the entry endpoint.\"\"\"\n", @@ -241,7 +240,7 @@ " \"chunk_MB\": round(v.dtype.itemsize * int(np.prod(ch)) / 1e6, 3) if ch else None,\n", " \"codec\": v.encoding.get(\"compression\"),\n", " \"shuffle\": bool(v.encoding.get(\"shuffle\"))})\n", - " return pd.DataFrame(rows).sort_values(\"MB\", ascending=False)" + " return pd.DataFrame(rows).sort_values(\"MB\", ascending=False)\n" ] }, { @@ -251,15 +250,17 @@ "source": [ "## METimage (VII) level 1B radiances\n", "\n", - "The full walkthrough. Each later instrument repeats these same five steps.\n", + "The full walkthrough. Later instruments repeat the discovery โ†’ range check โ†’\n", + "lazy open โ†’ layout steps. METimage also covers read-amplification timing and\n", + "VirtualiZarr/kerchunk references.\n", "\n", "First, the newest granule and its package contents. Expect one `.nc` and two\n", - "XML sidecars: the zip exists to carry those sidecars." + "XML sidecars: the zip exists to carry those sidecars.\n" ] }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 19, "id": "184c83eb", "metadata": {}, "outputs": [ @@ -267,8 +268,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "W_XX-EUMETSAT-Darmstadt,SAT,SGA1-VII-1B-RAD_C_EUMT_20260916205124_G_O_20260916200259_20260916200400_C_N_T__ \n", - " files: ['W_XX-EUMETSAT-Darmstadt,SAT,SGA1-VII-1B-RAD_C_EUMT_20260916205124_G_O_20260916200259_20260916200400_C_N_T__.nc', 'EOPMetadata.xml', 'manifest.xml']\n" + "W_XX-EUMETSAT-Darmstadt,SAT,SGA1-VII-1B-RAD_C_EUMT_20260922162541_G_O_20260922160859_20260922161000_C_N_T__ \n", + " files: ['W_XX-EUMETSAT-Darmstadt,SAT,SGA1-VII-1B-RAD_C_EUMT_20260922162541_G_O_20260922160859_20260922161000_C_N_T__.nc', 'EOPMetadata.xml', 'manifest.xml']\n" ] } ], @@ -290,7 +291,7 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 20, "id": "11775c38", "metadata": {}, "outputs": [ @@ -308,7 +309,7 @@ "True" ] }, - "execution_count": 17, + "execution_count": 20, "metadata": {}, "output_type": "execute_result" } @@ -324,13 +325,13 @@ "source": [ "Open the granule over HTTP. This reads metadata by range request and defers\n", "everything else, so it should finish in seconds even though the file is over\n", - "100 MB. Then pull a 64x64 slice of one radiance channel to confirm that\n", - "subsetting works and to feel the per-read latency." + "100 MB. The request log from ``open_remote`` shows how chatty that metadata\n", + "pass is.\n" ] }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 21, "id": "6da1635d", "metadata": { "scrolled": true @@ -340,8 +341,9 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 1.21 s, sys: 152 ms, total: 1.36 s\n", - "Wall time: 53.7 s\n" + "open_remote: 2 range requests, 8.4 MB transferred\n", + "CPU times: user 742 ms, sys: 18.8 ms, total: 761 ms\n", + "Wall time: 20.1 s\n" ] }, { @@ -899,17 +901,17 @@ "โ”‚ summary: VII/METimage L1B top of the atmosphere radiances\n", "โ”‚ doi: \n", "โ”‚ ... ...\n", - "โ”‚ sensing_start_time_utc: 2026-09-16 20:02:59.643\n", - "โ”‚ sensing_end_time_utc: 2026-09-16 20:04:00.134\n", + "โ”‚ sensing_start_time_utc: 2026-09-22 16:08:59.725\n", + "โ”‚ sensing_end_time_utc: 2026-09-22 16:10:00.215\n", "โ”‚ environment: Operational\n", "โ”‚ references: www.eumetsat.int\n", - "โ”‚ orbit_start: 5681\n", - "โ”‚ orbit_end: 5681\n", + "โ”‚ orbit_start: 5764\n", + "โ”‚ orbit_end: 5764\n", "โ”œโ”€โ”€ Group: /status\n", "โ”‚ โ”œโ”€โ”€ Group: /status/satellite\n", "โ”‚ โ”‚ Dimensions: ()\n", "โ”‚ โ”‚ Data variables: (12/24)\n", - "โ”‚ โ”‚ epoch_time_utc datetime64[ns] 8B ...\n", + "โ”‚ โ”‚ epoch_time_utc float64 8B ...\n", "โ”‚ โ”‚ semi_major_axis float64 8B ...\n", "โ”‚ โ”‚ eccentricity float64 8B ...\n", "โ”‚ โ”‚ inclination float64 8B ...\n", @@ -920,19 +922,19 @@ "โ”‚ โ”‚ yaw_error float64 8B ...\n", "โ”‚ โ”‚ roll_error float64 8B ...\n", "โ”‚ โ”‚ pitch_error float64 8B ...\n", - "โ”‚ โ”‚ leap_second_time_utc datetime64[ns] 8B ...\n", + "โ”‚ โ”‚ leap_second_time_utc float64 8B ...\n", "โ”‚ โ”‚ leap_second_value float32 4B ...\n", "โ”‚ โ”œโ”€โ”€ Group: /status/instrument\n", "โ”‚ โ”‚ Dimensions: (mode_items: 1)\n", "โ”‚ โ”‚ Dimensions without coordinates: mode_items\n", "โ”‚ โ”‚ Data variables:\n", - "โ”‚ โ”‚ mode_start_time_utc (mode_items) datetime64[ns] 8B ...\n", - "โ”‚ โ”‚ mode_end_time_utc (mode_items) datetime64[ns] 8B ...\n", + "โ”‚ โ”‚ mode_start_time_utc (mode_items) float64 8B ...\n", + "โ”‚ โ”‚ mode_end_time_utc (mode_items) float64 8B ...\n", "โ”‚ โ”‚ instrument_mode (mode_items) <U4 16B ...\n", "โ”‚ โ””โ”€โ”€ Group: /status/processing\n", "โ”‚ Dimensions: ()\n", "โ”‚ Data variables:\n", - "โ”‚ creation_time_utc datetime64[ns] 8B ...\n", + "โ”‚ creation_time_utc float64 8B ...\n", "โ”‚ Attributes:\n", "โ”‚ processor_name: VII_L1B\n", "โ”‚ processor_version: 1.0\n", @@ -1026,7 +1028,7 @@ " duration_of_data_missing float64 8B ...\n", " duration_of_data_degraded float64 8B ...\n", " Attributes:\n", - " overall_quality_flag: 0" + " overall_quality_flag: 0" ], "text/plain": [ "\n", @@ -1039,17 +1041,17 @@ "โ”‚ summary: VII/METimage L1B top of the atmosphere radiances\n", "โ”‚ doi: \n", "โ”‚ ... ...\n", - "โ”‚ sensing_start_time_utc: 2026-09-16 20:02:59.643\n", - "โ”‚ sensing_end_time_utc: 2026-09-16 20:04:00.134\n", + "โ”‚ sensing_start_time_utc: 2026-09-22 16:08:59.725\n", + "โ”‚ sensing_end_time_utc: 2026-09-22 16:10:00.215\n", "โ”‚ environment: Operational\n", "โ”‚ references: www.eumetsat.int\n", - "โ”‚ orbit_start: 5681\n", - "โ”‚ orbit_end: 5681\n", + "โ”‚ orbit_start: 5764\n", + "โ”‚ orbit_end: 5764\n", "โ”œโ”€โ”€ Group: /status\n", "โ”‚ โ”œโ”€โ”€ Group: /status/satellite\n", "โ”‚ โ”‚ Dimensions: ()\n", "โ”‚ โ”‚ Data variables: (12/24)\n", - "โ”‚ โ”‚ epoch_time_utc datetime64[ns] 8B ...\n", + "โ”‚ โ”‚ epoch_time_utc float64 8B ...\n", "โ”‚ โ”‚ semi_major_axis float64 8B ...\n", "โ”‚ โ”‚ eccentricity float64 8B ...\n", "โ”‚ โ”‚ inclination float64 8B ...\n", @@ -1060,19 +1062,19 @@ "โ”‚ โ”‚ yaw_error float64 8B ...\n", "โ”‚ โ”‚ roll_error float64 8B ...\n", "โ”‚ โ”‚ pitch_error float64 8B ...\n", - "โ”‚ โ”‚ leap_second_time_utc datetime64[ns] 8B ...\n", + "โ”‚ โ”‚ leap_second_time_utc float64 8B ...\n", "โ”‚ โ”‚ leap_second_value float32 4B ...\n", "โ”‚ โ”œโ”€โ”€ Group: /status/instrument\n", "โ”‚ โ”‚ Dimensions: (mode_items: 1)\n", "โ”‚ โ”‚ Dimensions without coordinates: mode_items\n", "โ”‚ โ”‚ Data variables:\n", - "โ”‚ โ”‚ mode_start_time_utc (mode_items) datetime64[ns] 8B ...\n", - "โ”‚ โ”‚ mode_end_time_utc (mode_items) datetime64[ns] 8B ...\n", + "โ”‚ โ”‚ mode_start_time_utc (mode_items) float64 8B ...\n", + "โ”‚ โ”‚ mode_end_time_utc (mode_items) float64 8B ...\n", "โ”‚ โ”‚ instrument_mode (mode_items) 256x amplification for 16 KB wanted\n", + "total : 12.6 MB of a 119 MB file\n" ] - }, - { - "data": { - "text/plain": [ - "True" - ] - }, - "execution_count": 29, - "metadata": {}, - "output_type": "execute_result" } ], "source": [ - "spec = PRODUCTS[\"MWS\"]\n", - "if spec.get(\"collection\"):\n", - " prod, entries = latest(spec[\"collection\"])\n", - " datafile = next(e for e in entries if not e.lower().endswith((\".xml\", \".txt\")))\n", - " url = entry_url(prod, datafile)\n", - "else:\n", - " url = spec[\"url\"] # direct test-data link\n", - " assert url, \"paste this product's link from the test-data page into PRODUCTS\"\n", + "import time\n", "\n", - "ranges_ok(url)" + "# Target mid-swath so we are not reading a corner that happens to be fill.\n", + "rad = tree[\"data/measurement_data\"].ds\n", + "var = next(v for v in rad.data_vars if v.startswith(\"vii_\"))\n", + "ny, nx = rad[var].shape[:2]\n", + "sl = (slice(ny // 2, ny // 2 + 64), slice(nx // 2, nx // 2 + 64))\n", + "chunks = rad[var].encoding.get(\"chunksizes\")\n", + "print(f\"variable : {var} {rad[var].shape} {rad[var].dtype}\")\n", + "print(f\"chunks : {chunks} codec={rad[var].encoding.get('compression')}\")\n", + "print(f\"file : {local.stat().st_size / 1e6:.0f} MB\\n\")\n", + "\n", + "# ---- local baseline -------------------------------------------------------\n", + "t0 = time.perf_counter()\n", + "a = xr.open_datatree(local, engine=\"h5netcdf\", phony_dims=\"access\",\n", + " decode_times=False)[\"data/measurement_data\"].ds[var][sl].values\n", + "t_local = time.perf_counter() - t0\n", + "wanted = a.nbytes\n", + "print(f\"local slice : {wanted / 1e3:.0f} KB in {t_local * 1e3:.0f} ms\\n\")\n", + "\n", + "# ---- remote: reopen with blockcache so miss_count is meaningful ----------\n", + "bs = 4 * 2**20\n", + "fs = fsspec.filesystem(\"https\", client_kwargs={\"headers\": auth()},\n", + " encoded=True, skip_instance_cache=True)\n", + "f = fs.open(url, mode=\"rb\", block_size=bs, cache_type=\"blockcache\",\n", + " size=remote_size(url))\n", + "\n", + "t0 = time.perf_counter()\n", + "rt = xr.open_datatree(f, engine=\"h5netcdf\", phony_dims=\"access\", decode_times=False)\n", + "t_open = time.perf_counter() - t0\n", + "open_bytes, open_reqs = f.cache.total_requested_bytes, f.cache.miss_count\n", + "\n", + "t0 = time.perf_counter()\n", + "b = rt[\"data/measurement_data\"].ds[var][sl].values\n", + "t_read = time.perf_counter() - t0\n", + "read_bytes = f.cache.total_requested_bytes - open_bytes\n", + "read_reqs = f.cache.miss_count - open_reqs\n", + "\n", + "print(f\"open : {open_bytes / 1e6:6.1f} MB in {open_reqs:3d} reads, {t_open:5.1f}s\")\n", + "print(f\"slice : {read_bytes / 1e6:6.1f} MB in {read_reqs:3d} reads, {t_read:5.1f}s\"\n", + " f\" -> {read_bytes / wanted:.0f}x amplification for {wanted / 1e3:.0f} KB wanted\")\n", + "print(f\"total : {(open_bytes + read_bytes) / 1e6:6.1f} MB of a \"\n", + " f\"{local.stat().st_size / 1e6:.0f} MB file\")\n", + "assert np.allclose(a, b, equal_nan=True)\n" + ] + }, + { + "cell_type": "markdown", + "id": "f65f8430-3766-4f7f-8dde-316d106e2038", + "metadata": {}, + "source": [ + "### VirtualiZarr โ†’ kerchunk references\n", + "\n", + "VirtualiZarr reads the local netCDF once and writes a small reference file.\n", + "Readers then use that sidecar (kerchunk JSON) to fetch only the bytes they\n", + "need from the remote URL โ€” without re-walking HDF5 metadata on every open.\n", + "\n" ] }, { "cell_type": "code", - "execution_count": 30, - "id": "6041c750", + "execution_count": 27, + "id": "89f82bc0-87a6-4637-9973-e90a83ce8db0", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "W_XX-EUMETSAT-Darmstadt,SAT,SGA1-MWS-1B-RAD_C_EUMT_20260916212055_G_O_20260916203857_20260916204158_O_N____.nc: 3 MB\n", - "compressed: 0 of 167 median chunk MB: 0.1075\n" + "data/metimage_l1b_sample.nc 119 MB\n", + "remote url: https://api.eumetsat.int/data/download/1.0.0/collections/EO%3AEUM%3ADAT%3A0464/products/W_XX-EUMETSAT-Darmstadt%2CSAT%2CSGA1-VII-1B-RAD_C_EUMT_20260922162541_G_O_20260922160859_20260922161000_C_N_T__/entry?name=W_XX-EUMETSAT-Darmstadt%2CSAT%2CSGA1-VII-1B-RAD_C_EUMT_20260922162541_G_O_20260922160859_20260922161000_C_N_T__.nc\n" ] - }, - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
variableshapeMBchunkschunk_MBcodecshuffle
130/data/calibration/mws_toa_radiance(80, 95, 24)1.46(28, 95, 24)0.511NoneFalse
108/data/calibration/mws_toa_brightness_temperature(80, 95, 24)1.46(28, 95, 24)0.511NoneFalse
136/data/measurement/mws_earth_view_counts(80, 95, 24)0.73(57, 95, 24)0.520NoneFalse
140/data/processing_information/mws_radiance_flag(80, 95, 24)0.18(80, 95, 24)0.182NoneFalse
162/data/processing_information/mws_brightnesstem...(80, 95, 24)0.18(80, 95, 24)0.182NoneFalse
102/data/navigation/mws_surface_type(80, 95, 2)0.12NoneNaNNoneFalse
99/data/navigation/mws_solar_azimuth_angle(80, 95)0.06NoneNaNNoneFalse
98/data/navigation/mws_satellite_zenith_angle(80, 95)0.06NoneNaNNoneFalse
104/data/navigation/mws_lat(80, 95)0.06NoneNaNNoneFalse
135/data/measurement/mws_earth_view_counts_os_stdev(80, 95, 2)0.06NoneNaNNoneFalse
\n", - "
" - ], - "text/plain": [ - " variable shape MB \\\n", - "130 /data/calibration/mws_toa_radiance (80, 95, 24) 1.46 \n", - "108 /data/calibration/mws_toa_brightness_temperature (80, 95, 24) 1.46 \n", - "136 /data/measurement/mws_earth_view_counts (80, 95, 24) 0.73 \n", - "140 /data/processing_information/mws_radiance_flag (80, 95, 24) 0.18 \n", - "162 /data/processing_information/mws_brightnesstem... (80, 95, 24) 0.18 \n", - "102 /data/navigation/mws_surface_type (80, 95, 2) 0.12 \n", - "99 /data/navigation/mws_solar_azimuth_angle (80, 95) 0.06 \n", - "98 /data/navigation/mws_satellite_zenith_angle (80, 95) 0.06 \n", - "104 /data/navigation/mws_lat (80, 95) 0.06 \n", - "135 /data/measurement/mws_earth_view_counts_os_stdev (80, 95, 2) 0.06 \n", - "\n", - " chunks chunk_MB codec shuffle \n", - "130 (28, 95, 24) 0.511 None False \n", - "108 (28, 95, 24) 0.511 None False \n", - "136 (57, 95, 24) 0.520 None False \n", - "140 (80, 95, 24) 0.182 None False \n", - "162 (80, 95, 24) 0.182 None False \n", - "102 None NaN None False \n", - "99 None NaN None False \n", - "98 None NaN None False \n", - "104 None NaN None False \n", - "135 None NaN None False " - ] - }, - "execution_count": 30, - "metadata": {}, - "output_type": "execute_result" } ], "source": [ - "tree = open_remote(url)\n", - "local = download(prod, datafile) if spec.get(\"collection\") else fetch(url)\n", - "L = layout(local)\n", - "print(\"compressed:\", int(L.codec.notna().sum()), \"of\", len(L),\n", - " \" median chunk MB:\", L.chunk_MB.median())\n", - "L.head(10)" - ] - }, - { - "cell_type": "markdown", - "id": "ad377891", - "metadata": {}, - "source": [ - "## Radio Occultation\n", - "\n", - "GRAS-2 has a Data Store collection, and its title says netCDF. The\n", - "`ranges_ok` call prints the first bytes as a check anyway: `894844` opens\n", - "like the others, `425546` spells BUFR, which has no internal chunking to\n", - "assess and would end this section early as its own finding.\n" + "# Index the local download, then rename chunk paths to the remote Data Store URL.\n", + "sample = DATA / \"metimage_l1b_sample.nc\"\n", + "if not sample.exists():\n", + " shutil.copy2(local, sample)\n", + "print(sample, f\"{sample.stat().st_size / 1e6:.0f} MB\")\n", + "print(\"remote url:\", url)\n" ] }, { "cell_type": "code", - "execution_count": 32, - "id": "b36ca262", + "execution_count": 28, + "id": "f9abbfdc-047e-4c39-a665-d0465964ea76", "metadata": {}, "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "W_XX-EUMETSAT-Darmstadt,SAT,SGA1-RO_-1B-BND_C_EUMT_20260916211844_G_O_20260916202243_20260916202900_O_N_G20 \n", - " files: ['W_XX-EUMETSAT-Darmstadt,SAT,SGA1-RO_-1B-BND_C_EUMT_20260916211844_G_O_20260916202243_20260916202900_O_N_G20.nc', 'EOPMetadata.xml', 'manifest.xml']\n", - "first bytes: 894844460d0a1a0a (894844... means HDF5/netCDF-4)\n", - "range status: {'head': 206, 'tail': 206}\n" - ] - }, { "data": { + "text/html": [ + "
\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
<xarray.Dataset> Size: 118MB\n",
+       "Dimensions:              (num_tie_points_alt: 140, num_tie_points_act: 394,\n",
+       "                          data/num_lines: 840, data/num_pixels: 3144,\n",
+       "                          data/num_scans: 35)\n",
+       "Dimensions without coordinates: num_tie_points_alt, num_tie_points_act,\n",
+       "                                data/num_lines, data/num_pixels, data/num_scans\n",
+       "Data variables: (12/32)\n",
+       "    latitude             (num_tie_points_alt, num_tie_points_act) int32 221kB ManifestArray<shape=(140, 394), dtype=int32, chunks=(1, 394)...\n",
+       "    longitude            (num_tie_points_alt, num_tie_points_act) uint32 221kB ManifestArray<shape=(140, 394), dtype=uint32, chunks=(1, 394...\n",
+       "    delta_lat_N_dem      (data/num_lines, data/num_pixels) int16 5MB Manifest...\n",
+       "    delta_lon_E_dem      (data/num_lines, data/num_pixels) int16 5MB Manifest...\n",
+       "    solar_zenith         (num_tie_points_alt, num_tie_points_act) uint32 221kB ManifestArray<shape=(140, 394), dtype=uint32, chunks=(1, 394...\n",
+       "    solar_azimuth        (num_tie_points_alt, num_tie_points_act) uint32 221kB ManifestArray<shape=(140, 394), dtype=uint32, chunks=(1, 394...\n",
+       "    ...                   ...\n",
+       "    vii_6725             (data/num_lines, data/num_pixels) uint16 5MB Manifes...\n",
+       "    vii_7325             (data/num_lines, data/num_pixels) uint16 5MB Manifes...\n",
+       "    vii_8540             (data/num_lines, data/num_pixels) uint16 5MB Manifes...\n",
+       "    vii_10690            (data/num_lines, data/num_pixels) uint16 5MB Manifes...\n",
+       "    vii_12020            (data/num_lines, data/num_pixels) uint16 5MB Manifes...\n",
+       "    vii_13345            (data/num_lines, data/num_pixels) uint16 5MB Manifes...
" + ], "text/plain": [ - "True" + " Size: 118MB\n", + "Dimensions: (num_tie_points_alt: 140, num_tie_points_act: 394,\n", + " data/num_lines: 840, data/num_pixels: 3144,\n", + " data/num_scans: 35)\n", + "Dimensions without coordinates: num_tie_points_alt, num_tie_points_act,\n", + " data/num_lines, data/num_pixels, data/num_scans\n", + "Data variables: (12/32)\n", + " latitude (num_tie_points_alt, num_tie_points_act) int32 221kB ManifestArray 1019x amplification for 16 KB wanted\n", - "total : 92.2 MB of a 119 MB file\n" + "7.22 MB sidecar for a 119 MB granule\n", + "CPU times: user 73.1 ms, sys: 979 ฮผs, total: 74.1 ms\n", + "Wall time: 11.6 s\n", + "vii_443 (64, 64) allclose True\n" ] } ], "source": [ - "import time, urllib.parse\n", - "from pathlib import Path\n", - "import xarray as xr, fsspec, requests\n", - "\n", - "LOCAL = Path(\"data/W_XX-EUMETSAT-Darmstadt,SAT,SGA1-VII-1B-RAD_C_EUMT_20260916205124_G_O_20260916200259_20260916200400_C_N_T__.nc\")\n", - "enc = lambda s: urllib.parse.quote(s, safe=\"\")\n", - "\n", - "pid = LOCAL.name[:-3] # product id = filename minus .nc\n", - "base = (\"https://api.eumetsat.int/data/download/1.0.0/collections/\"\n", - " f\"{enc('EO:EUM:DAT:0464')}/products/{enc(pid)}\")\n", - "URL = f\"{base}/entry?name={enc(LOCAL.name)}\"\n", - "\n", - "def remote_size(u):\n", - " r = requests.get(u, headers={**auth(), \"Range\": \"bytes=0-0\"}, stream=True, timeout=60); r.close()\n", - " return int(r.headers[\"Content-Range\"].split(\"/\")[-1]) if r.status_code == 206 \\\n", - " else int(r.headers[\"Content-Length\"])\n", - "\n", - "# ---- pick the target variable from the local copy -------------------------\n", - "lt = xr.open_datatree(LOCAL, engine=\"h5netcdf\", phony_dims=\"access\", decode_times=False)\n", - "rad = lt[\"data/measurement_data\"].ds\n", - "var = \"vii_443\"\n", - "ny, nx = rad[var].shape[:2]\n", - "sl = (slice(ny//2, ny//2+64), slice(nx//2, nx//2+64)) # mid-swath, not a corner\n", - "chunks = rad[var].encoding.get(\"chunksizes\")\n", - "print(f\"variable : {var} {rad[var].shape} {rad[var].dtype}\")\n", - "print(f\"chunks : {chunks} codec={rad[var].encoding.get('compression')}\")\n", - "print(f\"file : {LOCAL.stat().st_size/1e6:.0f} MB\\n\")\n", - "\n", - "# ---- local baseline -------------------------------------------------------\n", - "t0 = time.perf_counter(); a = rad[var][sl].values; t_local = time.perf_counter()-t0\n", - "wanted = a.nbytes\n", - "print(f\"local slice : {wanted/1e3:.0f} KB in {t_local*1e3:.0f} ms\\n\")\n", - "\n", - "bs = 8*1024*1024\n", - "fs = fsspec.filesystem(\"https\", client_kwargs={\"headers\": auth()},\n", - " encoded=True, skip_instance_cache=True)\n", - "f = fs.open(URL, block_size=bs, size=remote_size(URL))\n", - "\n", - "t0 = time.perf_counter()\n", - "rt = xr.open_datatree(f, engine=\"h5netcdf\", phony_dims=\"access\", decode_times=False)\n", - "t_open = time.perf_counter() - t0\n", - "open_bytes, open_reqs = f.cache.total_requested_bytes, f.cache.miss_count\n", + "# Point every chunk at the remote URL (byte offsets stay the same).\n", + "vds_remote = vds.vz.rename_paths(url)\n", "\n", - "t0 = time.perf_counter()\n", - "b = rt[\"data/measurement_data\"].ds[var][sl].values\n", - "t_read = time.perf_counter() - t0\n", - "read_bytes = f.cache.total_requested_bytes - open_bytes\n", - "read_reqs = f.cache.miss_count - open_reqs\n", + "refs_path = Path(\"metimage_refs.json\")\n", + "vds_remote.vz.to_kerchunk(str(refs_path), format=\"json\")\n", + "print(f\"{refs_path.stat().st_size / 1e6:.2f} MB sidecar for a \"\n", + " f\"{sample.stat().st_size / 1e6:.0f} MB granule\")\n", "\n", - "print(f\"open : {open_bytes/1e6:6.1f} MB in {open_reqs:3d} reads, {t_open:5.1f}s\")\n", - "print(f\"slice : {read_bytes/1e6:6.1f} MB in {read_reqs:3d} reads, {t_read:5.1f}s\"\n", - " f\" -> {read_bytes/wanted:.0f}x amplification for {wanted/1e3:.0f} KB wanted\")\n", - "print(f\"total : {(open_bytes+read_bytes)/1e6:6.1f} MB of a \"\n", - " f\"{LOCAL.stat().st_size/1e6:.0f} MB file\")" + "# Zarr v3 needs the reference FS and its HTTPS target both asynchronous.\n", + "fs = fsspec.filesystem(\n", + " \"reference\",\n", + " fo=str(refs_path),\n", + " remote_protocol=\"https\",\n", + " asynchronous=True,\n", + " remote_options={\"headers\": auth(), \"encoded\": True, \"asynchronous\": True},\n", + ")\n", + "ds = xr.open_dataset(\n", + " fs.get_mapper(\"\"),\n", + " engine=\"zarr\",\n", + " consolidated=False,\n", + " decode_times=False,\n", + ")\n", + "%time sample_remote = ds[var][sl].values\n", + "print(var, sample_remote.shape, \"allclose\",\n", + " np.allclose(a, sample_remote, equal_nan=True))\n" ] }, { "cell_type": "markdown", - "id": "f65f8430-3766-4f7f-8dde-316d106e2038", + "id": "9ded0d0e", "metadata": {}, "source": [ - "### Making Virtual Zarr" + "## MWS\n", + "\n", + "Microwave sounder, 24 channels, small granules. The same steps usually run in\n", + "under a minute end to end." ] }, { "cell_type": "code", - "execution_count": 3, - "id": "89f82bc0-87a6-4637-9973-e90a83ce8db0", + "execution_count": 30, + "id": "39b5e427", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "https://api.eumetsat.int/data/download/1.0.0/collections/EO%3AEUM%3ADAT%3A0464/products/W_XX-EUMETSAT-Darmstadt%2CSAT%2CSGA1-VII-1B-RAD_C_EUMT_20260916205124_G_O_20260916200259_20260916200400_C_N_T__/entry?name=W_XX-EUMETSAT-Darmstadt%2CSAT%2CSGA1-VII-1B-RAD_C_EUMT_20260916205124_G_O_20260916200259_20260916200400_C_N_T__.nc\n" + "W_XX-EUMETSAT-Darmstadt,SAT,SGA1-MWS-1B-RAD_C_EUMT_20260922161338_G_O_20260922160559_20260922160857_O_N____ \n", + " files: ['W_XX-EUMETSAT-Darmstadt,SAT,SGA1-MWS-1B-RAD_C_EUMT_20260922161338_G_O_20260922160559_20260922160857_O_N____.nc', 'EOPMetadata.xml', 'manifest.xml']\n", + "first bytes: 894844460d0a1a0a (894844... means HDF5/netCDF-4)\n", + "range status: {'head': 206, 'tail': 206}\n" ] + }, + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 30, + "metadata": {}, + "output_type": "execute_result" } ], "source": [ - "import urllib.parse, eumdac\n", - "from pathlib import Path\n", - "\n", - "LOCAL = Path(\"data/W_XX-EUMETSAT-Darmstadt,SAT,SGA1-VII-1B-RAD_C_EUMT_20260916205124_G_O_20260916200259_20260916200400_C_N_T__.nc\")\n", - "enc = lambda s: urllib.parse.quote(s, safe=\"\")\n", - "\n", - "KEY, SECRET = Path.home().joinpath(\".eumdac\", \"credentials\").read_text().strip().split(\",\")\n", - "token = eumdac.AccessToken((KEY, SECRET))\n", + "spec = PRODUCTS[\"MWS\"]\n", + "if spec.get(\"collection\"):\n", + " prod, entries = latest(spec[\"collection\"])\n", + " datafile = next(e for e in entries if not e.lower().endswith((\".xml\", \".txt\")))\n", + " url = entry_url(prod, datafile)\n", + "else:\n", + " url = spec[\"url\"] # direct test-data link\n", + " assert url, \"paste this product's link from the test-data page into PRODUCTS\"\n", "\n", - "pid = LOCAL.name[:-3]\n", - "url = (f\"https://api.eumetsat.int/data/download/1.0.0/collections/{enc('EO:EUM:DAT:0464')}\"\n", - " f\"/products/{enc(pid)}/entry?name={enc(LOCAL.name)}\")\n", - "print(url)" + "ranges_ok(url)" ] }, { "cell_type": "code", - "execution_count": 7, - "id": "f9abbfdc-047e-4c39-a665-d0465964ea76", + "execution_count": 31, + "id": "6041c750", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "True\n" + "open_remote: 1 range requests, 4.2 MB transferred\n", + "W_XX-EUMETSAT-Darmstadt,SAT,SGA1-MWS-1B-RAD_C_EUMT_20260922161338_G_O_20260922160559_20260922160857_O_N____.nc: 3 MB\n", + "compressed: 0 of 167 median chunk MB: 0.1065\n" ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
variableshapeMBchunkschunk_MBcodecshuffle
130/data/calibration/mws_toa_radiance(79, 95, 24)1.44(28, 95, 24)0.511NoneFalse
108/data/calibration/mws_toa_brightness_temperature(79, 95, 24)1.44(28, 95, 24)0.511NoneFalse
136/data/measurement/mws_earth_view_counts(79, 95, 24)0.72(57, 95, 24)0.520NoneFalse
141/data/processing_information/mws_radiance_flag(79, 95, 24)0.18(79, 95, 24)0.180NoneFalse
162/data/processing_information/mws_brightnesstem...(79, 95, 24)0.18(79, 95, 24)0.180NoneFalse
99/data/navigation/mws_surface_type(79, 95, 2)0.12NoneNaNNoneFalse
96/data/navigation/mws_solar_azimuth_angle(79, 95)0.06NoneNaNNoneFalse
95/data/navigation/mws_satellite_zenith_angle(79, 95)0.06NoneNaNNoneFalse
135/data/measurement/mws_earth_view_counts_os_stdev(79, 95, 2)0.06NoneNaNNoneFalse
94/data/navigation/mws_solar_zenith_angle(79, 95)0.06NoneNaNNoneFalse
\n", + "
" + ], + "text/plain": [ + " variable shape MB \\\n", + "130 /data/calibration/mws_toa_radiance (79, 95, 24) 1.44 \n", + "108 /data/calibration/mws_toa_brightness_temperature (79, 95, 24) 1.44 \n", + "136 /data/measurement/mws_earth_view_counts (79, 95, 24) 0.72 \n", + "141 /data/processing_information/mws_radiance_flag (79, 95, 24) 0.18 \n", + "162 /data/processing_information/mws_brightnesstem... (79, 95, 24) 0.18 \n", + "99 /data/navigation/mws_surface_type (79, 95, 2) 0.12 \n", + "96 /data/navigation/mws_solar_azimuth_angle (79, 95) 0.06 \n", + "95 /data/navigation/mws_satellite_zenith_angle (79, 95) 0.06 \n", + "135 /data/measurement/mws_earth_view_counts_os_stdev (79, 95, 2) 0.06 \n", + "94 /data/navigation/mws_solar_zenith_angle (79, 95) 0.06 \n", + "\n", + " chunks chunk_MB codec shuffle \n", + "130 (28, 95, 24) 0.511 None False \n", + "108 (28, 95, 24) 0.511 None False \n", + "136 (57, 95, 24) 0.520 None False \n", + "141 (79, 95, 24) 0.180 None False \n", + "162 (79, 95, 24) 0.180 None False \n", + "99 None NaN None False \n", + "96 None NaN None False \n", + "95 None NaN None False \n", + "135 None NaN None False \n", + "94 None NaN None False " + ] + }, + "execution_count": 31, + "metadata": {}, + "output_type": "execute_result" } ], "source": [ - "import shutil\n", - "from pathlib import Path\n", - "\n", - "root = Path(\"data\").resolve()\n", - "src = root / \"W_XX-EUMETSAT-Darmstadt,SAT,SGA1-VII-1B-RAD_C_EUMT_20260916205124_G_O_20260916200259_20260916200400_C_N_T__.nc\"\n", - "LOCAL = root / \"metimage_l1b_sample.nc\"\n", - "if not LOCAL.exists():\n", - " shutil.copy2(src, LOCAL) # or src.link_to / os.link to avoid duplicating 119 MB\n", - "print(LOCAL.exists())" + "tree = open_remote(url)\n", + "local = download(prod, datafile) if spec.get(\"collection\") else fetch(url)\n", + "L = layout(local)\n", + "print(\"compressed:\", int(L.codec.notna().sum()), \"of\", len(L),\n", + " \" median chunk MB:\", L.chunk_MB.median())\n", + "L.head(10)" ] }, { - "cell_type": "code", - "execution_count": 8, - "id": "73929cc4-b88f-499e-afeb-7be304a5145b", + "cell_type": "markdown", + "id": "ad377891", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - " Size: 118MB\n", - "Dimensions: (num_tie_points_alt: 140, num_tie_points_act: 394,\n", - " data/num_lines: 840, data/num_pixels: 3144,\n", - " data/num_scans: 35)\n", - "Dimensions without coordinates: num_tie_points_alt, num_tie_points_act,\n", - " data/num_lines, data/num_pixels, data/num_scans\n", - "Data variables: (12/32)\n", - " latitude (num_tie_points_alt, num_tie_points_act) int32 221kB ManifestArray Date: Tue, 22 Sep 2026 12:05:09 -0500 Subject: [PATCH 4/5] Update metopsg_cloud_native.ipynb From 7b51bfda5b4b91479e42019b58c20568654f9bb3 Mon Sep 17 00:00:00 2001 From: Harshini Date: Tue, 22 Sep 2026 12:06:34 -0500 Subject: [PATCH 5/5] Update pip install command in Jupyter notebook --- eumetsat_metopsg/metopsg_cloud_native.ipynb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/eumetsat_metopsg/metopsg_cloud_native.ipynb b/eumetsat_metopsg/metopsg_cloud_native.ipynb index 17d220c..b153663 100644 --- a/eumetsat_metopsg/metopsg_cloud_native.ipynb +++ b/eumetsat_metopsg/metopsg_cloud_native.ipynb @@ -52,7 +52,8 @@ "output_type": "stream", "text": [ "Note: you may need to restart the kernel to use updated packages.\n" - ] + ], + "name": "stdout" } ], "source": [