Skip to content

Commit 693588f

Browse files
ehinmanehinman
authored
Merge pull request #183 from ehinman/add-waterdata-infrastructure
Add waterdata infrastructure
2 parents 2c8419c + c575447 commit 693588f

18 files changed

Lines changed: 2808 additions & 504 deletions

‎.github/workflows/python-package.yml‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,10 @@ jobs:
1717
matrix:
1818
os: [ubuntu-latest, windows-latest]
1919
python-version: [3.8, 3.9, '3.10', 3.11, 3.12]
20+
exclude:
21+
- os: ubuntu-latest
22+
python-version: 3.8
23+
2024

2125
steps:
2226
- uses: actions/checkout@eef61447b9ff4aafe5dcd4e0bbf5d482be7e7871
@@ -36,7 +40,5 @@ jobs:
3640
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
3741
- name: Test with pytest and report coverage
3842
run: |
39-
cd tests
40-
coverage run -m pytest
43+
coverage run -m pytest tests/
4144
coverage report -m
42-
cd ..

‎NEWS.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
**11/24/2025:** `dataretrieval` is pleased to offer a new module, `waterdata`, which gives users access USGS's modernized [Water Data APIs](https://api.waterdata.usgs.gov/). The Water Data API endpoints include daily values, instantaneous values, field measurements (modernized groundwater levels service), time series metadata, and discrete water quality data from the Samples database. Though there will be a period of overlap, the functions within `waterdata` will eventually replace the `nwis` module, which currently provides access to the legacy [NWIS Water Services](https://waterservices.usgs.gov/). More example workflows and functions coming soon. Check `help(waterdata)` for more information.
2+
3+
**09/03/2024:** The groundwater levels service has switched endpoints, and `dataretrieval` was updated accordingly in [`v1.0.10`](https://github.com/DOI-USGS/dataretrieval-python/releases/tag/v1.0.10). Older versions using the discontinued endpoint will return 503 errors for `nwis.get_gwlevels` or the `service='gwlevels'` argument. Visit [Water Data For the Nation](https://waterdata.usgs.gov/blog/wdfn-waterservices-2024/) for more information.
4+
5+
**03/01/2024:** USGS data availability and format have changed on Water Quality Portal (WQP). Since March 2024, data obtained from WQP legacy profiles will not include new USGS data or recent updates to existing data. All USGS data (up to and beyond March 2024) are available using the new WQP beta services. You can access the beta services by setting `legacy=False` in the functions in the `wqp` module.
6+
7+
To view the status of changes in data availability and code functionality, visit: https://doi-usgs.github.io/dataRetrieval/articles/Status.html

‎README.md‎

Lines changed: 217 additions & 77 deletions
Original file line numberDiff line numberDiff line change
@@ -4,123 +4,263 @@
44
![Conda Version](https://img.shields.io/conda/v/conda-forge/dataretrieval)
55
![Downloads](https://static.pepy.tech/badge/dataretrieval)
66

7-
:warning: USGS data availability and format have changed on Water Quality Portal (WQP). Since March 2024, data obtained from WQP legacy profiles will not include new USGS data or recent updates to existing data. All USGS data (up to and beyond March 2024) are available using the new WQP beta services. You can access the beta services by setting `legacy=False` in the functions in the `wqp` module.
7+
## Latest Announcements
88

9-
To view the status of changes in data availability and code functionality, visit: https://doi-usgs.github.io/dataRetrieval/articles/Status.html
9+
:mega: **11/24/2025:** `dataretrieval` now features the new `waterdata` module,
10+
which provides access to USGS's modernized [Water Data
11+
APIs](https://api.waterdata.usgs.gov/). The Water Data API endpoints include
12+
daily values, instantaneous values, field measurements, time series metadata,
13+
and discrete water quality data from the Samples database. This new module will
14+
eventually replace the `nwis` module, which provides access to the legacy [NWIS
15+
Water Services](https://waterservices.usgs.gov/).
1016

11-
:mega: **09/03/2024:** The groundwater levels service has switched endpoints, and `dataretrieval` was updated accordingly in [`v1.0.10`](https://github.com/DOI-USGS/dataretrieval-python/releases/tag/v1.0.10). Older versions using the discontinued endpoint will return 503 errors for `nwis.get_gwlevels` or the `service='gwlevels'` argument. Visit [Water Data For the Nation](https://waterdata.usgs.gov/blog/wdfn-waterservices-2024/) for more information.
17+
**Important:** Users of the Water Data APIs are strongly encouraged to obtain an
18+
API key for higher rate limits and greater access to USGS data. [Register for
19+
an API key](https://api.waterdata.usgs.gov/signup/) and set it as an
20+
environment variable:
21+
22+
```python
23+
import os
24+
os.environ["API_USGS_PAT"] = "your_api_key_here"
25+
```
26+
27+
Check out the [NEWS](NEWS.md) file for all updates and announcements.
1228

1329
## What is dataretrieval?
14-
`dataretrieval` was created to simplify the process of loading hydrologic data into the Python environment.
15-
Like the original R version [`dataRetrieval`](https://github.com/DOI-USGS/dataRetrieval),
16-
it is designed to retrieve the major data types of U.S. Geological Survey (USGS) hydrology
17-
data that are available on the Web, as well as data from the Water
18-
Quality Portal (WQP), which currently houses water quality data from the
19-
Environmental Protection Agency (EPA), U.S. Department of Agriculture
20-
(USDA), and USGS. Direct USGS data is obtained from a service called the
21-
National Water Information System (NWIS).
2230

23-
Note that the python version is not a direct port of the original: it attempts to reproduce the functionality of the R package,
24-
though its organization and interface often differ.
31+
`dataretrieval` simplifies the process of loading hydrologic data into Python.
32+
Like the original R version
33+
[`dataRetrieval`](https://github.com/DOI-USGS/dataRetrieval), it retrieves major
34+
U.S. Geological Survey (USGS) hydrology data types available on the Web, as well
35+
as data from the Water Quality Portal (WQP) and Network Linked Data Index
36+
(NLDI).
2537

26-
If there's a hydrologic or environmental data portal that you'd like dataretrieval to
27-
work with, raise it as an [issue](https://github.com/USGS-python/dataretrieval/issues).
38+
## Installation
2839

29-
Here's an example using `dataretrieval` to retrieve data from the National Water Information System (NWIS).
40+
Install dataretrieval using pip:
3041

31-
```python
32-
# first import the functions for downloading data from NWIS
33-
import dataretrieval.nwis as nwis
42+
```bash
43+
pip install dataretrieval
44+
```
45+
46+
Or using conda:
47+
48+
```bash
49+
conda install -c conda-forge dataretrieval
50+
```
51+
52+
## Usage Examples
53+
54+
### Water Data API (Recommended - Modern USGS Data)
3455

35-
# specify the USGS site code for which we want data.
36-
site = '03339000'
56+
The `waterdata` module provides access to modern USGS Water Data APIs.
3757

38-
# get instantaneous values (iv)
39-
df = nwis.get_record(sites=site, service='iv', start='2017-12-31', end='2018-01-01')
58+
The example below retrieves daily streamflow data for a specific monitoring
59+
location for water year 2025, where a "/" between two dates in the "time"
60+
input argument indicates a desired date range:
4061

41-
# get basic info about the site
42-
df2 = nwis.get_record(sites=site, service='site')
62+
```python
63+
import dataretrieval.waterdata as waterdata
64+
65+
# Get daily streamflow data (returns DataFrame and metadata)
66+
df, metadata = waterdata.get_daily(
67+
monitoring_location_id='USGS-01646500',
68+
parameter_code='00060', # Discharge
69+
time='2024-10-01/2025-09-30'
70+
)
71+
72+
print(f"Retrieved {len(df)} records")
73+
print(f"Site: {df['monitoring_location_id'].iloc[0]}")
74+
print(f"Mean discharge: {df['value'].mean():.2f} {df['unit_of_measure'].iloc[0]}")
4375
```
44-
Services available from NWIS include:
45-
- instantaneous values (iv)
46-
- daily values (dv)
47-
- statistics (stat)
48-
- site info (site)
49-
- discharge peaks (peaks)
50-
- discharge measurements (measurements)
51-
52-
Water quality data are available from:
53-
- [Samples](https://waterdata.usgs.gov/download-samples/#dataProfile=site) - Discrete USGS water quality data only
54-
- [Water Quality Portal](https://www.waterqualitydata.us/) - Discrete water quality data from USGS and EPA. Older data are available in the legacy WQX version 2 format; all data are available in the beta WQX3.0 format.
55-
56-
To access the full functionality available from NWIS web services, nwis.get record appends any additional kwargs into the REST request. For example, this function call:
76+
Fetch daily discharge data for multiple sites from a start date to present
77+
using the following code:
78+
5779
```python
58-
nwis.get_record(sites='03339000', service='dv', start='2017-12-31', parameterCd='00060')
80+
df, metadata = waterdata.get_daily(
81+
monitoring_location_id=["USGS-13018750","USGS-13013650"],
82+
parameter_code='00060',
83+
time='2024-10-01/..'
84+
)
85+
86+
print(f"Retrieved {len(df)} records")
5987
```
60-
...will download daily data with the parameter code 00060 (discharge).
88+
The following example downloads location information for all monitoring
89+
locations that are categorized as stream sites in the state of Maryland:
6190

62-
## Accessing the "Internal" NWIS
63-
If you're connected to the USGS network, dataretrieval call pull from the internal (non-public) NWIS interface.
64-
Most dataretrieval functions pass kwargs directly to NWIS's REST API, which provides simple access to internal data; simply specify "access='3'".
65-
For example
6691
```python
67-
nwis.get_record(sites='05404147',service='iv', start='2021-01-01', end='2021-3-01', access='3')
92+
# Get monitoring location information
93+
locations, metadata = waterdata.get_monitoring_locations(
94+
state_name='Maryland',
95+
site_type_code='ST' # Stream sites
96+
)
97+
98+
print(f"Found {len(locations)} stream monitoring locations in Maryland")
6899
```
100+
Visit the
101+
[API Reference](https://doi-usgs.github.io/dataretrieval-python/reference/waterdata.html)
102+
for more information and examples on available services and input parameters.
69103

70-
More services and documentation to come!
104+
**NEW:** This new module implements
105+
[logging](https://docs.python.org/3/howto/logging.html#logging-basic-tutorial)
106+
in which users can view the URL requests sent to the USGS Water Data APIs
107+
and the number of requests they have remaining each hour. These messages can
108+
be helpful for troubleshooting and support. To enable logging in your python
109+
console or notebook:
71110

72-
## Quick start
111+
```python
112+
import logging
113+
logging.basicConfig(level=logging.INFO)
114+
```
115+
To log messages to a file, you can specify a filename in the
116+
`basicConfig` call:
73117

74-
dataretrieval can be installed using pip:
75-
76-
$ python3 -m pip install -U dataretrieval
118+
```python
119+
logging.basicConfig(filename='waterdata.log', level=logging.INFO)
120+
```
77121

78-
or conda:
122+
### NWIS Legacy Services (Deprecated but still functional)
79123

80-
$ conda install -c conda-forge dataretrieval
124+
The `nwis` module accesses legacy NWIS Water Services:
81125

82-
More examples of use are include in [`demos`](https://github.com/USGS-python/dataretrieval/tree/main/demos).
126+
```python
127+
import dataretrieval.nwis as nwis
83128

84-
## Issue tracker
129+
# Get site information
130+
info, metadata = nwis.get_info(sites='01646500')
131+
132+
print(f"Site name: {info['station_nm'].iloc[0]}")
133+
134+
# Get daily values
135+
dv, metadata = nwis.get_dv(
136+
sites='01646500',
137+
start='2024-10-01',
138+
end='2024-10-02',
139+
parameterCd='00060',
140+
)
141+
142+
print(f"Retrieved {len(dv)} daily values")
143+
```
85144

86-
Please report any bugs and enhancement ideas using the dataretrieval issue
87-
tracker:
145+
### Water Quality Portal (WQP)
88146

89-
https://github.com/USGS-python/dataretrieval/issues
147+
Access water quality data from multiple agencies:
90148

91-
Feel free to also ask questions on the tracker.
149+
```python
150+
import dataretrieval.wqp as wqp
92151

152+
# Find water quality monitoring sites
153+
sites = wqp.what_sites(
154+
statecode='US:55', # Wisconsin
155+
siteType='Stream'
156+
)
93157

94-
## Contributing
158+
print(f"Found {len(sites)} stream monitoring sites in Wisconsin")
159+
160+
# Get water quality results
161+
results = wqp.get_results(
162+
siteid='USGS-05427718',
163+
characteristicName='Temperature, water'
164+
)
95165

96-
Any help in testing, development, documentation and other tasks is welcome.
97-
For more details, see the file [CONTRIBUTING.md](CONTRIBUTING.md).
166+
print(f"Retrieved {len(results)} temperature measurements")
167+
```
98168

169+
### Network Linked Data Index (NLDI)
99170

100-
## Need help?
171+
Discover and navigate hydrologic networks:
101172

102-
The Water Mission Area of the USGS supports the development and maintenance of `dataretrieval`. Any questions can be directed to the Computational Tools team at
103-
comptools@usgs.gov.
173+
```python
174+
import dataretrieval.nldi as nldi
104175

105-
Resources are available primarily for maintenance and responding to user questions.
106-
Priorities on the development of new features are determined by the `dataretrieval` development team.
176+
# Get watershed basin for a stream reach
177+
basin = nldi.get_basin(
178+
feature_source='comid',
179+
feature_id='13293474' # NHD reach identifier
180+
)
107181

182+
print(f"Basin contains {len(basin)} feature(s)")
183+
184+
# Find upstream flowlines
185+
flowlines = nldi.get_flowlines(
186+
feature_source='comid',
187+
feature_id='13293474',
188+
navigation_mode='UT', # Upstream tributaries
189+
distance=50 # km
190+
)
191+
192+
print(f"Found {len(flowlines)} upstream tributaries within 50km")
193+
```
194+
195+
## Available Data Services
196+
197+
### Modern USGS Water Data APIs (Recommended)
198+
- **Daily values**: Daily statistical summaries (mean, min, max)
199+
- **Field measurements**: Discrete measurements from field visits
200+
- **Monitoring locations**: Site information and metadata
201+
- **Time series metadata**: Information about available data parameters
202+
- **Latest daily values**: Most recent daily statistical summary data
203+
- **Latest instantaneous values**: Most recent high-frequency continuous data
204+
- **Samples data**: Discrete USGS water quality data
205+
- **Instantaneous values** (*COMING SOON*): High-frequency continuous data
206+
207+
### Legacy NWIS Services (Deprecated)
208+
- **Daily values (dv)**: Legacy daily statistical data
209+
- **Instantaneous values (iv)**: Legacy continuous data
210+
- **Site info (site)**: Basic site information
211+
- **Statistics (stat)**: Statistical summaries
212+
- **Discharge peaks (peaks)**: Annual peak discharge events
213+
- **Discharge measurements (measurements)**: Direct flow measurements
214+
215+
### Water Quality Portal
216+
- **Results**: Water quality analytical results from USGS, EPA, and other agencies
217+
- **Sites**: Monitoring location information
218+
- **Organizations**: Data provider information
219+
- **Projects**: Sampling project details
220+
221+
### Network Linked Data Index (NLDI)
222+
- **Basin delineation**: Watershed boundaries for any point
223+
- **Flow navigation**: Upstream/downstream network traversal
224+
- **Feature discovery**: Find monitoring sites, dams, and other features
225+
- **Hydrologic connectivity**: Link data across the stream network
226+
227+
## More Examples
228+
229+
Explore additional examples in the
230+
[`demos`](https://github.com/USGS-python/dataretrieval/tree/main/demos)
231+
directory, including Jupyter notebooks demonstrating advanced usage patterns.
232+
233+
## Getting Help
234+
235+
- **Issue tracker**: Report bugs and request features at https://github.com/USGS-python/dataretrieval/issues
236+
- **Documentation**: Full API documentation available in the source code docstrings
237+
238+
## Contributing
239+
240+
Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for
241+
development guidelines.
108242

109243
## Acknowledgments
110-
This material is partially based upon work supported by the National Science Foundation (NSF) under award 1931297.
111-
Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF.
244+
245+
This material is partially based upon work supported by the National Science
246+
Foundation (NSF) under award 1931297. Any opinions, findings, conclusions, or
247+
recommendations expressed in this material are those of the authors and do not
248+
necessarily reflect the views of the NSF.
112249

113250
## Disclaimer
114251

115-
This software is preliminary or provisional and is subject to revision.
116-
It is being provided to meet the need for timely best science.
117-
The software has not received final approval by the U.S. Geological Survey (USGS).
118-
No warranty, expressed or implied, is made by the USGS or the U.S. Government as to the functionality of the software and related material nor shall the fact of release constitute any such warranty.
119-
The software is provided on the condition that neither the USGS nor the U.S. Government shall be held liable for any damages resulting from the authorized or unauthorized use of the software.
252+
This software is preliminary or provisional and is subject to revision. It is
253+
being provided to meet the need for timely best science. The software has not
254+
received final approval by the U.S. Geological Survey (USGS). No warranty,
255+
expressed or implied, is made by the USGS or the U.S. Government as to the
256+
functionality of the software and related material nor shall the fact of release
257+
constitute any such warranty. The software is provided on the condition that
258+
neither the USGS nor the U.S. Government shall be held liable for any damages
259+
resulting from the authorized or unauthorized use of the software.
120260

121261
## Citation
122262

123-
Hodson, T.O., Hariharan, J.A., Black, S., and Horsburgh, J.S., 2023, dataretrieval (Python): a Python package for discovering
124-
and retrieving water data available from U.S. federal hydrologic web services:
125-
U.S. Geological Survey software release,
126-
https://doi.org/10.5066/P94I5TX3.
263+
Hodson, T.O., Hariharan, J.A., Black, S., and Horsburgh, J.S., 2023,
264+
dataretrieval (Python): a Python package for discovering and retrieving water
265+
data available from U.S. federal hydrologic web services: U.S. Geological Survey
266+
software release, https://doi.org/10.5066/P94I5TX3.

‎dataretrieval/__init__.py‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,16 @@
11
from importlib.metadata import PackageNotFoundError, version
22

3+
try:
4+
__version__ = version("dataretrieval")
5+
except PackageNotFoundError:
6+
__version__ = "version-unknown"
7+
38
from dataretrieval.nadp import *
9+
from dataretrieval.nldi import *
410
from dataretrieval.nwis import *
511
from dataretrieval.samples import *
612
from dataretrieval.streamstats import *
713
from dataretrieval.utils import *
814
from dataretrieval.waterdata import *
915
from dataretrieval.waterwatch import *
1016
from dataretrieval.wqp import *
11-
12-
try:
13-
__version__ = version("dataretrieval")
14-
except PackageNotFoundError:
15-
__version__ = "version-unknown"

0 commit comments

Comments
 (0)