add inputs to timeseries function, edit new vignette

Author: ehinman

dataretrieval/waterdata/api.py

@@ -691,6 +691,8 @@ def get_time_series_metadata(
     parameter_name: Optional[Union[str, List[str]]] = None,
     properties: Optional[Union[str, List[str]]] = None,
     statistic_id: Optional[Union[str, List[str]]] = None,
+    hydrologic_unit_code: Optional[Union[str, List[str]]] = None,
+    state_name: Optional[Union[str, List[str]]] = None,
     last_modified: Optional[Union[str, List[str]]] = None,
     begin: Optional[Union[str, List[str]]] = None,
     end: Optional[Union[str, List[str]]] = None,
@@ -742,6 +744,17 @@ def get_time_series_metadata(
         Example codes include 00001 (max), 00002 (min), and 00003 (mean).
         A complete list of codes and their descriptions can be found at
         https://help.waterdata.usgs.gov/code/stat_cd_nm_query?stat_nm_cd=%25&fmt=html.
+    hydrologic_unit_code : string or list of strings, optional
+        The United States is divided and sub-divided into successively smaller
+        hydrologic units which are classified into four levels: regions,
+        sub-regions, accounting units, and cataloging units. The hydrologic
+        units are arranged within each other, from the smallest (cataloging units)
+        to the largest (regions). Each hydrologic unit is identified by a unique
+        hydrologic unit code (HUC) consisting of two to eight digits based on the
+        four levels of classification in the hydrologic unit system.
+    state_name : string or list of strings, optional
+        The name of the state or state equivalent in which the monitoring location
+        is located.
     last_modified : string, optional
         The last time a record was refreshed in our database. This may happen
         due to regular operational processes and does not necessarily indicate

demos/WaterData_demo.ipynb

@@ -8,7 +8,7 @@
     "# Using the `waterdata` module to pull data from the USGS Water Data APIs\n",
     "The `waterdata` module will eventually replace the `nwis` module for accessing USGS water data. It leverages the [Water Data APIs](https://api.waterdata.usgs.gov/) to download metadata, daily values, and instantaneous values. \n",
     "\n",
-    "While the specifics of this transition timeline are hazy, it is advised to switch to the new functions as soon as possible to reduce unexpected interruptions in your workflow.\n",
+    "While the specifics of this transition timeline are opaque, it is advised to switch to the new functions as soon as possible to reduce unexpected interruptions in your workflow.\n",
     "\n",
     "As always, please report any issues you encounter on our [Issues](https://github.com/DOI-USGS/dataretrieval-python/issues) page. If you have questions or need help, please reach out to us at comptools@usgs.gov."
    ]
@@ -29,7 +29,7 @@
     "``` \n",
     "Note that the environment variable name is `API_USGS_PAT`, which stands for \"API USGS Personal Access Token\".\n",
     "\n",
-    "If you'd like a more permanent repository-specific solution, you can use the `python-dotenv` package to read your API key from a `.env` file in your repository root directory, like this:\n",
+    "If you'd like a more permanent, repository-specific solution, you can use the `python-dotenv` package to read your API key from a `.env` file in your repository root directory, like this:\n",
     "\n",
     "```python\n",
     "!pip install python-dotenv # only run this line once to install the package in your environment\n",
@@ -55,7 +55,7 @@
     "These functions retrieve metadata tables that can be used to refine your data requests.\n",
     "\n",
     "- `get_reference_table()` - Not sure which parameter code you're looking for, or which hydrologic unit your study area is in? This function will help you find the right input values for the data endpoints to retrieve the information you want.\n",
-    "- `get_codes()` - Similar to `get_reference_table()`, this function retrieves dataframes containing available input values that correspond to the Samples database.\n",
+    "- `get_codes()` - Similar to `get_reference_table()`, this function retrieves dataframes containing available input values that correspond to the Samples water quality database.\n",
     "\n",
     "### Data endpoints\n",
     "- `get_daily()` - Daily values for monitoring locations, parameters, stat codes, and more.\n",
@@ -68,6 +68,17 @@
     "- `get_samples()` - Discrete water quality sample results for monitoring locations, observed properties, and more."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "19b5aebf",
+   "metadata": {},
+   "source": [
+    "### A few key tips\n",
+    "- You'll notice that each of the data functions have many unique inputs you can specify. **DO NOT** specify too many! Specify *just enough* inputs to return what you need. But do not provide redundant geographical or parameter information as this may slow down your query and lead to errors.\n",
+    "- Each function returns a Tuple, containing a dataframe and a Metadata class. If you have `geopandas` installed in your environment, the dataframe will be a `GeoDataFrame` with a geometry included. If you do not have `geopandas`, the dataframe will be a `pandas` dataframe with the geometry contained in a coordinates column. The Metadata object contains information about your query, like the query url.\n",
+    "- If you do not want to return the `geometry` column, use the input `skip_geometry=True`."
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "68591b52",
@@ -84,7 +95,11 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "import pandas as pd\n",
     "from IPython.display import display\n",
+    "from datetime import datetime, timedelta\n",
+    "from datetime import date\n",
+    "from dateutil.relativedelta import relativedelta\n",
     "from dataretrieval import waterdata"
    ]
   },
@@ -101,51 +116,99 @@
   },
   {
    "cell_type": "markdown",
-   "id": "176c665b",
+   "id": "1e0eab77",
    "metadata": {},
    "source": [
-    "What is this `metadata` element? Let's take a look:"
+    "Let's say we want to find all parameter codes relating to streamflow discharge. We can use some string matching to find applicable codes."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "30b1b052",
+   "id": "665ccb23",
    "metadata": {},
    "outputs": [],
    "source": [
-    "metadata"
+    "streamflow_pcodes = pcodes[pcodes['parameter_name'].str.contains('streamflow|discharge', case=False, na=False)]\n",
+    "display(streamflow_pcodes[['parameter_code_id', 'parameter_name']])"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "1e0eab77",
+   "id": "d9487ee4",
    "metadata": {},
    "source": [
-    "All of these functions return Tuples containing a dataframe and a metadata element containing descriptors about the request made. This `BaseMetadata` object contains the request URL.\n",
+    "Interesting that there are so many different streamflow-related parameter codes! Going on experience, let's use the most common one, `00060`, which is \"Discharge, cubic feet per second\".\n",
     "\n",
-    "Let's say we want to find all parameter codes relating to streamflow discharge. We can use some string matching to find applicable codes."
+    "Now that we know which parameter code we want to use, let's find all the stream monitoring locations that have recent discharge data and at least 10 years of daily values in the state of Nebraska. We will use the `waterdata.get_time_series_metadata()` function to suss out which sites fit the bill. This function will return a row for each *timeseries* that matches our inputs. It doesn't contain the daily discharge values themselves, just information *about* that timeseries."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "70ee1da9",
+   "metadata": {},
+   "source": [
+    "First, let's get our expected date range in order. Note that the `waterdata` functions are capable of taking in bounded and unbounded date and datetime ranges. In this case, we want the start date of the discharge timeseries to be no more recent than 10 years ago, and we want the end date of the timeseries to be from at most a week ago."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "665ccb23",
+   "id": "57e2c93a",
    "metadata": {},
    "outputs": [],
    "source": [
-    "streamflow_pcodes = pcodes[pcodes['parameter_name'].str.contains('streamflow|discharge', case=False, na=False)]\n",
-    "display(streamflow_pcodes[['parameter_code_id', 'parameter_name']])"
+    "ten_years_ago =(date.today() - relativedelta(years=10))\n",
+    "one_week_ago = (datetime.now() - timedelta(days=7)).date()"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "d9487ee4",
+   "id": "2cd98164",
+   "metadata": {},
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a901f5fa",
    "metadata": {},
+   "outputs": [],
    "source": [
-    "Interesting that there are so many different streamflow-related parameter codes! Going on experience, let's use the most common one, `00060`, which is \"Discharge, cubic feet per second\".\n",
-    "\n",
-    "Now that we know which parameter code we want to use, let's find all the stream monitoring locations that have recent discharge data and at least 10 years of daily values in the state of Nebraska. "
+    "NE_discharge,_ = waterdata.get_time_series_metadata(\n",
+    "    state_name=\"Nebraska\",\n",
+    "    parameter_code='00060',\n",
+    "    begin=f\"1700-01-01/{ten_years_ago}\",\n",
+    "    end=f\"{one_week_ago}/..\",\n",
+    "    skip_geometry=True\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8809a98d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "display(NE_discharge.sort_values(\"monitoring_location_id\").head())\n",
+    "print(f\"There are {len(NE_discharge['monitoring_location_id'].unique())} sites with recent discharge data available in the state of Nebraska\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8f464470",
+   "metadata": {},
+   "source": [
+    "In the dataframe above, we are looking at 5 timeseries returned, ordered by monitoring location. You can also see that the first two rows show two different kinds of discharge for the same monitoring location: a mean daily discharge timeseries (with statistic id 00003, which represents \"mean\") and an instantaneous discharge timeseries (with statistic id 00011, which represents \"points\" or \"instantaneous\" values). Look closely and you may also notice that the `parent_timeseries_id` column for daily mean discharge matches the `time_series_id` for the instantaneous discharge. This is because once instantaneous measurements began at the site, they were used to calculate the daily mean."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "452b8830",
+   "metadata": {},
+   "source": [
+    "Now that we know which sites have recent discharge data, let's find stream sites and plot them on a map. We will use the `waterdata.get_monitoring_locations()` function to grab more metadata about these sites. Even though we have a list of monitoring location IDs from the timeseries function, it's faster to use the `state_name` argument to return all stream sites, and then filter down to the ones we're interested in."
    ]
   },
   {
@@ -155,8 +218,104 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "NE_locations,_ = waterdata.get_monitoring_locations(state_name=\"Nebraska\", site_type_code=\"ST\")\n",
-    "display(NE_locations.head())"
+    "NE_locations,_ = waterdata.get_monitoring_locations(\n",
+    "    state_name=\"Nebraska\",\n",
+    "    site_type_code=\"ST\"\n",
+    "    )\n",
+    "\n",
+    "NE_locations_discharge = NE_locations.loc[NE_locations['monitoring_location_id'].isin(NE_discharge['monitoring_location_id'].unique().tolist())]\n",
+    "display(NE_locations_discharge[[\"monitoring_location_id\", \"monitoring_location_name\", \"hydrologic_unit_code\"]].head())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f0fe5c4e",
+   "metadata": {},
+   "source": [
+    "If you have `geopandas` installed, the function will return a `GeoDataFrame` with a `geometry` column containing the monitoring locations' coordinates. If you don't have `geopandas` installed, it will return a regular `pandas` DataFrame with coordinate columns instead. Let's take a look at the site locations using `gpd.explore()`. Hover over the site points to see all the columns returned from `waterdata.get_monitoring_locations()`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "659b19a5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "NE_locations_discharge.set_crs(crs=\"WGS84\").explore()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f1d92784",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ne_sites = NE_locations['monitoring_location_id'].to_list()\n",
+    "print(len(ne_sites))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "897ca5e1",
+   "metadata": {},
+   "source": [
+    "We cannot feed 1700+ monitoring location ID's into the time series metadata function, so we will need to break this list up into smaller chunks and loop through them."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "58307318",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chunk_size=50\n",
+    "chunks = [ne_sites[i:i + chunk_size] for i in range(0, len(ne_sites), chunk_size)]\n",
+    "len(chunks)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "89379400",
+   "metadata": {},
+   "source": [
+    "Now, we will loop through each chunk of sites and pull timeseries information for sites with discharge data from the past week and a timeseries that is at least 10 years old."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "52a67e9e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dfs = pd.DataFrame()\n",
+    "for site_group in chunks:\n",
+    "        try:\n",
+    "            timeseries_info,_ = waterdata.get_time_series_metadata(\n",
+    "                monitoring_location_id=site_group,\n",
+    "                parameter_code='00060',\n",
+    "                begin=f\"1700-01-01/{ten_years_ago}\",\n",
+    "                end=f\"{one_week_ago}/..\",\n",
+    "                skip_geometry=True\n",
+    "            )\n",
+    "            if not timeseries_info.empty:\n",
+    "                dfs = pd.concat([dfs, timeseries_info])\n",
+    "        except Exception as e:\n",
+    "            # Log & continue; you can also implement retries here\n",
+    "            print(f\"Batch failed (size={len(site_group)}): {e}\")\n",
+    "\n",
+    "display(dfs.head())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d1b01ca3",
+   "metadata": {},
+   "source": [
+    "One thing you might notice is that this dataframe returns a `state_name` column!"
    ]
   }
  ],