Skip to content

Commit 4155a8d

Browse files
KristinaRiemerdlebauer
KristinaRiemer
authored and
dlebauer
committed
Changes to get weather tutorial working (#74)
* Clean up R code and expand explanations up to High Resolution section * Update curl section description, fix curl flag typo * Fix link and picture insertion * Improve query API section * made a few changes re #74 that I will submit as PR * removed extra unused variable names * Fix format of objectives section and add to prereqs * Remove extra backslash from URL querying section * Remove entire 'your turn' section * Create combined plot * Change date range for weather data to April through Aug 2018 * Keep only the plot for wind speed
1 parent 88ea83a commit 4155a8d

1 file changed

Lines changed: 66 additions & 104 deletions

File tree

‎sensors/01-meteorological-data.Rmd‎

Lines changed: 66 additions & 104 deletions
Original file line numberDiff line numberDiff line change
@@ -3,24 +3,16 @@ title: "01-meteorological-data.Rmd"
33
output: html_document
44
---
55

6-
```{r met-setup, include=FALSE}
7-
knitr::opts_chunk$set(echo = FALSE, cache = TRUE)
6+
Objectives:
87

9-
api_url <- "https://terraref.ncsa.illinois.edu/clowder/api"
10-
output_dir <- file.path(tempdir(), "downloads")
11-
dir.create(output_dir, showWarnings = FALSE, recursive = TRUE)
8+
* This tutorial will walk through the steps required to access meteorological data from the Maricopa Agricultural Center.
129

13-
```
10+
Pre-requisites:
1411

15-
## The Maricopa Weather Station
12+
* Need to have R packages tidyverse, jsonlite, and convertr installed.
13+
* Need to have an internet connection.
1614

17-
Locating the weather station via the geodashboard:
18-
19-
TODO: screenshots / tutorials
20-
21-
Geodashboard:
22-
23-
[Sample raw meteorological (1/s) data on Clowder](https://terraref.ncsa.illinois.edu/clowder/files/588ba5474f0c06726acfcace?dataset=587fc7444f0cd67174e7a92d&space=57e42cd44f0cff4b58dd3eea)
15+
## The Maricopa Weather Station
2416

2517
## Meteorological data formats
2618

@@ -36,26 +28,16 @@ Geodashboard:
3628

3729
| CF standard-name | units | bety | isimip | cruncep | narr | ameriflux |
3830
|:------------------------------------------|:------|:-------------|:-------------|:--------|:------|:----------|
39-
| **air_temperature** | K | airT | tasAdjust | tair | air | TA (C) |
40-
| air_temperature_max | K | | tasmaxAdjust | NA | tmax | |
41-
| air_temperature_min | K | | tasminAdjust | NA | tmin | |
42-
| **air_pressure** | Pa | air_pressure | | | | PRESS (KPa) |
31+
| air_temperature | K | airT | tasAdjust | tair | air | TA (C) |
32+
| air_pressure | Pa | air_pressure | | | | PRESS (KPa) |
4333
| mole_fraction_of_carbon_dioxide_in_air | mol/mol | | | | | CO2 |
44-
| moisture_content_of_soil_layer | kg m-2 | | | | | |
45-
| soil_temperature | K | soilT | | | | TS1 *(NOT DONE)* |
4634
| relative_humidity | % | relative_humidity | rhurs | NA | rhum | RH |
47-
| **specific_humidity** | 1 | specific_humidity | NA | qair | shum | CALC(RH) |
48-
| water_vapor_saturation_deficit | Pa | VPD | | | | VPD *(NOT DONE)* |
49-
| **surface_downwelling_longwave_flux_in_air** | W m-2 | same | rldsAdjust | lwdown | dlwrf | Rgl |
50-
| **surface_downwelling_shortwave_flux_in_air**| W m-2 |solar_radiation|rsdsAdjust| swdown | dswrf | Rg |
5135
| surface_downwelling_photosynthetic_photon_flux_in_air | mol m-2 s-1 | PAR | | | | PAR *(NOT DONE)* |
52-
| **precipitation_flux** | kg m-2 s-1 | cccc | prAdjust | rain | acpc | PREC (mm/s) |
36+
| precipitation_flux | kg m-2 s-1 | cccc | prAdjust | rain | acpc | PREC (mm/s) |
5337
| | degrees | wind_direction | | | | WD |
5438
| wind_speed | m/s | Wspd | | | | WS |
55-
| **eastward_wind** | m/s | eastward_wind | | | | CALC(WS+WD) |
56-
| **northward_wind** | m/s | northward_wind | | | | CALC(WS+WD) |
5739

58-
* preferred variables indicated in bold
40+
5941
* variable names are from [MsTMIP](http://nacp.ornl.gov/MsTMIP_variables.shtml).
6042
* standard_name is CF-convention standard names
6143
* units can be converted by udunits, so these can vary (e.g. the time denominator may change with time frequency of inputs)
@@ -68,31 +50,32 @@ The key is to process each type of met data (site, reanalysis, forecast, climate
6850

6951
### The Geostreams Database
7052

71-
!(schema)[https://cloud.githubusercontent.com/assets/9286213/16991300/b2f2b09a-4e60-11e6-96b7-8b63c3d1f995.jpg]
53+
![Schema](https://cloud.githubusercontent.com/assets/9286213/16991300/b2f2b09a-4e60-11e6-96b7-8b63c3d1f995.jpg)
7254

7355
#### Querying the API
7456

75-
key query terms:
76-
77-
* /sensors
78-
* sensor_name=''
79-
* /streams
80-
*
81-
* /datapoints
82-
* &since=YYYY-MM-DD
83-
* &until=YYYY-MM-DD
84-
* stream_id=
57+
The data can be accessed using a URL. You can find data for a particular station,
58+
a certain sensor from a station, or the datapoints from a sensor. Below are
59+
the URLs for the UA-MAC AZMET weather station, a certain sensor at that station,
60+
and datapoints from that sensor for April 2018 through to the first day of
61+
August 2018.
8562

86-
How to find the stream_id from a particular station or sensor:
63+
* Station: https://terraref.ncsa.illinois.edu/clowder/api/geostreams/sensors?sensor_name=UA-MAC+AZMET+Weather+Station
64+
* Sensor: https://terraref.ncsa.illinois.edu/clowder/api/geostreams/sensors/438/streams
65+
* Datapoints: https://terraref.ncsa.illinois.edu/clowder/api/geostreams/datapoints?stream_id=46431&since=2018-04-01&until=2018-08-01
8766

88-
1. find streams on "UA-MAC AZMET Weather Station" https://terraref.ncsa.illinois.edu/clowder/api/geostreams/sensors?sensor_name=UA-MAC AZMET Weather Station
89-
2. take sensor id from that response and find the streams: https://terraref.ncsa.illinois.edu/clowder/api/geostreams/sensors/438/streams
90-
3. take stream id from that and use in the datapoints query: https://terraref.ncsa.illinois.edu/clowder/api/geostreams/datapoints?stream_id=46431&since=2017-01-02&until=2017-01-31
67+
All URLs have the same beginning (https://terraref.ncsa.illinois.edu/clowder/api/geostreams),
68+
then additional information is added for each type of data as shown below.
9169

70+
* Station: /sensors/sensor_name=[name]
71+
* Sensor: /sensors/[sensor number]/streams
72+
* Datapoints: /datapoints?stream_id=[datapoints number]&since=[start date]&until=[end date]
9273

93-
#### Locations with met data
74+
Possible sensor numbers for a station are found on the page for that station under
75+
"id:", and then datapoints numbers are found on the sensor page under "stream_id:".
9476

95-
These are some of the locations with met data:
77+
The table belows lists the names of some stations that have available
78+
meteorological data and associated stream ids.
9679

9780
| stream id | name |
9881
|------------|------------------------------------------|
@@ -144,88 +127,60 @@ The data represent 5 minute summaries aggregated from 1/s observations.
144127

145128
#### Using Curl
146129

147-
First, this is what the API looks like as a URL. Try pasting it into your browser
130+
First, below is what the API looks like as a URL. Try pasting it into your browser.
148131

149-
https://terraref.ncsa.illinois.edu/clowder/api/geostreams/datapoints?stream_id=46431&since=2017-01-02&until=2017-01-31
132+
https://terraref.ncsa.illinois.edu/clowder/api/geostreams/datapoints?stream_id=46431&since=2018-04-01&until=2018-08-01
150133

151-
This is how you can automatically download the met data to a local file:
134+
These data are for one stream within a chosen time period. They can be
135+
automatically downloaded into a file on your computer by typing the following
136+
into the command line. This uses the URL from above and the new file is named
137+
spectra.json.
152138

153139
```{sh eval=FALSE}
154-
curl -O spectra.json -X GET https://terraref.ncsa.illinois.edu/clowder/api/geostreams/datapoints?stream_id=46431&since=2017-01-02&until=2017-01-31
140+
curl -o spectra.json -X GET https://terraref.ncsa.illinois.edu/clowder/api/geostreams/datapoints?stream_id=46431&since=2018-04-01&until=2018-08-01
155141
```
156142

157-
And this is how you can access the data in R:
143+
#### Using R
144+
145+
The following code sets the defaults for showing R code.
146+
```{r met-setup}
147+
knitr::opts_chunk$set(cache = FALSE, message = FALSE)
148+
```
158149

150+
And this is how you can access the same data in R. This uses the jsonlite R package
151+
and desired URL to pull the data in. The data is in a dataframe with two nested
152+
dataframes, called `properties` and `geometries`.
159153

160154
```{r met-geostream}
161155
library(dplyr)
162156
library(ggplot2)
163157
library(jsonlite)
158+
library(lubridate)
164159
165-
mac_weather.list <- jsonlite::fromJSON('https://terraref.ncsa.illinois.edu/clowder/api/geostreams/datapoints?stream_id=46431&since=2017-01-02&until=2017-01-31', flatten = FALSE)
166-
167-
# change time to human-readable
168-
mac_weather <- mac_weather.list$properties %>%
169-
mutate(time = lubridate::ymd_hms(mac_weather.list$end_time))
160+
weather_all <- fromJSON('https://terraref.ncsa.illinois.edu/clowder/api/geostreams/datapoints?stream_id=46431&since=2018-04-01&until=2018-08-01', flatten = FALSE)
170161
```
171162

172-
### Using
163+
The `geometries` dataframe is then pulled out from these data, which contains
164+
the datapoints from this stream. This is combined with a transformed version of the
165+
end of the time period from the stream.
173166

174-
## Weather Summary
167+
```{r met-datapoints}
168+
weather_data <- weather_all$properties %>%
169+
mutate(time = ymd_hms(weather_all$end_time))
170+
```
175171

172+
### Weather Plots
176173

177-
#### Wind Speed
174+
Create time series plot for one of the eight variables, wind speed, in the
175+
newly created dataframe.
178176

179177
```{r weather}
180178
theme_set(ggthemes::theme_few())
181-
ggplot(data = mac_weather) +
182-
geom_point(aes(x = time, y = wind_speed), size = 0.1)
183-
```
184-
185-
#### Rainfall
186-
187-
```{r precipitation}
188-
ggplot(data = mac_weather) +
189-
geom_point(aes(x = time, y = precipitation_rate), size = 0.1)
179+
ggplot(data = weather_data) +
180+
geom_point(aes(x = time, y = wind_speed), size = 0.7) +
181+
labs(x = "Day", y = "Wind speed (m/s)")
190182
```
191183

192-
#### Your turn!
193-
194-
Try the following:
195-
* Convert temperature in K to C
196-
* what other unit conversions would be useful?
197-
* plot two metrics of solar radiation flux.
198-
* What is the difference between these?
199-
200-
Here is one way you might convert the given units into ones appropriate for the scale that you are working on:
201-
202-
```{r convert-units}
203-
# from https://raw.githubusercontent.com/PecanProject/pecan/master/models/biocro/R/met2model.BIOCRO.R
204-
library(convertr)
205-
library(lubridate)
206-
met <- mac_weather %>%
207-
transmute(date = time,
208-
par = surface_downwelling_photosynthetic_photon_flux_in_air,
209-
temperature = convert(air_temperature, "K", "degC"),
210-
rh = relative_humidity,
211-
ws = wind_speed,
212-
precip = convert(precipitation_rate, "1/s", "1/min") * 5 )
213-
214-
## your turn: summarize by day
215-
216-
ggplot(data = met %>% filter(date > ymd("2016-08-01") & date < ymd("2016-12-01"))) +
217-
geom_line(aes(date, precip))
218-
```
219-
220-
```{r precip-mm-m}
221-
ggplot(data = met %>% filter(date > ymd("2016-08-01") & date < ymd("2016-10-01"))) +
222-
geom_point(aes(date, precip))
223-
```
224-
225-
#### A notice on reuseable code
226-
227-
Did you see what we just did? We wrote some custom code to convert the units of our meteorological data. Wouldn't it be nice if that were available as a library function? In fact, it is. That code was from the function `cf2biocro()` in the PEcAn.BIOCRO package. It converts standard meteorological data to the format that the user wants.
228-
229184
# High resolution data (1/s) + spectroradiometer
230185

231186
This higher resolution weather data can be used for VNIR calibration, for example. But at 1/s it is very large!
@@ -234,6 +189,13 @@ This higher resolution weather data can be used for VNIR calibration, for exampl
234189

235190
Here we will download the files using the Clowder API, but note that if you have access to the filesystem (on www.workbench.terraref.org or globus, you can directly access the data in the `sites/ua-mac/Level_1/EnvironmentLogger`. Folder
236191

192+
```{r met-setup2}
193+
knitr::opts_chunk$set(eval = FALSE)
194+
api_url <- "https://terraref.ncsa.illinois.edu/clowder/api"
195+
output_dir <- file.path(tempdir(), "downloads")
196+
dir.create(output_dir, showWarnings = FALSE, recursive = TRUE)
197+
```
198+
237199
```{r query-clowder}
238200
library(jsonlite)
239201
library(magrittr)

0 commit comments

Comments
 (0)