updates - got simulated data tutorial to work; output to /docs

Author: dlebauer

_bookdown.yml

@@ -1,4 +1,5 @@
 book_filename: "terraref-tutorials"
+output_dir: "book-output"
 language:
   ui:
     chapter_name: "Chapter "

traits/00-BETYdb-getting-started.Rmd

@@ -11,23 +11,23 @@ The TERRA Ref program uses the BETYdb database and web application software to s
 
 ### BETYdb: database software and web application
 
+The TERRA REF trait database (terraref.ncsa.illinois.edu/bety) uses the BETYdb data schema (structure) and web application.
 The BETYdb software is actively used and developed by the [TERRA Reference](terraref.org) program as well as by the [PEcAn project](pecanproject.org).
 
 For more information about BETYdb, see the following:
 
 * BETYdb documentation (available via the web application under 'Docs')
-  * _Data Access_
+  * _Data Access_: how to access data
   * _Data Entry Workflow:_ how to add data to the database
   * _BETYdb Technical Documentation_ is written for advanced users and website and database administrators who may also be interested in the [full database schema](betydb.org/schemas)
 * BETYdb: A Yield, Trait and Ecosystem Service Database Applied to Second Generation Bioenergy Feedstocks. ([LeBauer et al, 2017](dx.doi.org/10.1111/gcbb.12420))
 
-The TERRA REF trait database (terraref.ncsa.illinois.edu/bety) uses the BETYdb data schema (structure) and web application.
 There are at least a half-dozen other databases using the BETYdb software that these exercises will work with, though the results will depend on the available data.
 The first, betydb.org is described in LeBauer et al, 2017.
 Others are listed in the 'distributed BETYdb' section of the technical documentation.
 
-One database, terraref.ncsa.illinois.edu/terra-test, houses a simulated dataset that is used in [lesson 1: A simulated data set](../traits/01-simulated-sorghum.Rmd) and does not require an account to access the data.
-BETYdb is only designed to keep the primary data private. Metadata such as field management and experimental design are available if the url is public.
+When there is a public-facing website, BETYdb is only designed to keep its trait and yield data private.
+Metadata such as field management and experimental design are available if the url is public.
 
 ## Getting an account for the TERRA trait database
 
@@ -40,22 +40,30 @@ TODO add signup info from handout
 
 ## First steps: download data from web interface
 
-TODO add steps to download csv from the web interface
+* Point your browser to terraref.ncsa.illinois.edu/bety
+* login 
+* enter "NDVI" in the search box
+* on the next page you will see the results of this search 
+  * if you want all of the data, including data that has not gone through QA/QC, make sure to check the 'include unchecked records' option
+* in the upper right, you will see a button that will allow you to download the search results as a CSV file. Click it. Open the file in a text editor or spreadsheet program and review its contents.
 
-Note that the web interface only provides a core set of data. More complex queries, such as those in the [Agronomic metadata](../traits/04-agronomic-metadata.Rmd)
+Note that the web interface only provides a core set of data and limited meta-data. To access all of the data within BETYdb, it is necessary to search and merge multiple tables. More complex queries, such as those in the [Agronomic metadata](../traits/04-agronomic-metadata.Rmd).
 
 ## Advanced: Using URLs to construct Queries
 
 The first step toward reproducible pipelines is to automate the process of searching the database and returning results. This is one of the key roles of an Application programming interface, or 'API'. You can learn to use the API in less than 20 minutes, starting now. 
 
 ### What is an API?
 
-An API is ...
+An API is an 'Application Programming Interface'. An API is a way that you and your software can connect to and access data. 
+
+All of our databases have web interfaces for humans to browse as well as APIs that are constructed as URLs. 
+
 
 ### Using Your API key to Connect
 
 An API key is like a password. It allows you to access data, and should be kept private. 
-Therefore, we are not going to put it in code that we share. The one exception is the key 9999999999999999999999999999999999999999 that will allow you to access metadata tables (all tables except _traits_ and _yields_). 
+Therefore, we are not going to put it in code that we share. The one exception is the key 9999999999999999999999999999999999999999 that will allow you to access metadata tables (all tables except _traits_ and _yields_). It will also allow you to access all of the simulated data in the terraref.ncsa.illinois.edu/bety-test database.
 
 A common way of handling private API keys is to place it in a text file in your home directory. 
 Don't put it in a project directory where it might be inadvertently shared.
@@ -66,9 +74,17 @@ Here is how to find and save your API key:
 * copy the api key that was sent when you registered into the file
 * file --> save as '~/.betykey'
 
-Equivalently in R `r writeLines('9999999999999999999999999999999999999999', con = '~/.betykey')` or at the command line `sh echo 9999999999999999999999999999999999999999 > ~/.betykey`
+For the public key, you can call this file `~/.betykey_public`. 
+
+### Components of a URL query
+
+
+* base url: `terraref.ncsa.illinois.edu/bety`
+* path to the api: `/api/beta`
+* api endpoint: `/search` or `traits` or `sites`. For BETYdb, these are the names of database tables. 
+* Query parameters: `genus=Sorghum`
+* Authentication: `key=9999999999999999999999999999999999999999` is the public key for the TERRA REF traits database. 
 
-For the purposes of the tutorial, you can assign it to the `betykey` variable in the console window.
 
 ### Constructing a URL query
 
@@ -86,27 +102,68 @@ First, lets construct a query by putting together a URL.
   * `name=~height` where the variable name contains 'height'
 5. This is your complete query:
   * `terraref.ncsa.illinois.edu/bety/api/beta/variables?type=trait&name=~height&key=9999999999999999999999999999999999999999`
+  * it will query all variables that are type trait and have 'height' in the name
+  * Does it return the expected values?
 
-**Your Turn**
+
+#### Your Turn
+
+> What will the URL https://terraref.ncsa.illinois.edu/bety/api/beta/species?genus=Sorghum&key=9999999999999999999999999999999999999999 return?
 
 > write a URL that will query the database for sites with "Field Scanner" in the name field. Hint: combine two terms with a `+` as in `Field+Scanner`
 
 What do you see? Do you think that this is all of the records? What happens if you add `&limit=none`? 
 
+### Our first Query 
+
+#### Shell
+
+```sh
+wget -O sorghum.json \\ # -O names the output file 
+   "https://terraref.ncsa.illinois.edu/bety/api/beta/species?genus=Sorghum&key=999999999999999999999999999999999999
+9999"
+```
+
+If you want to write the query without exposing the key in plain text, you can construct it thus:
+
+```sh
+wget -O sorghum.json \\
+    "https://terraref.ncsa.illinois.edu/bety/api/beta/species?genus=Sorghum&key=`cat ~/.betykey_public`"
+```
+
+> What does `cat ~/.betykey_public` do?
+
+> How can you look at the files?
+
+
+#### R - using the jsonlite package
+
+```{r text-api}
+sorghum.json <- readLines(
+  paste0("https://terraref.ncsa.illinois.edu/bety/api/beta/species?genus=Sorghum&key=", 
+         readLines('~/.betykey')))
+
+## print(sorghum.json) 
+## not a particularly useful format
+## lets convert to a data frame
+sorghum <- jsonlite::fromJSON(sorghum.json)
+```
+
 ## Using the R traits package to query the database
 
 The rOpenSci traits package makes it easier to query the TERRA REF trait database, or any database that uses BETYdb software.
 
-First, make sure we have the latest version 
+First, make sure we have the latest version from the terraref fork of the repository on github. (you can install using the standard `install.packages('traits')` but I can't promise everything will work.
+
+### Install the package
 
 ```{r install_traits, echo=FALSE}
-if(packageVersion("traits") == '0.2.0'){
-  devtools::install_github('ropensci/traits')
-}
+devtools::install_github('terraref/traits')
 ```
 
+Now, we can load the packages that we will need to get started.
 
-```{r setup}
+```{r 00-setup}
 library(traits)
 knitr::opts_chunk$set(echo = FALSE, cache = TRUE)
 library(ggplot2)
@@ -126,49 +183,63 @@ library(dplyr)
 writeLines('9999999999999999999999999999999999999999', 
            con = '~/.betykey_public')
 ```
-```{r traits}
-terraref_test_url <- 
-public_betykey <- 
 
-```
+#### R - using the traits package
 
-### Our first Query 
+The R traits package is an API 'client'. It does two important things:
+1. It makes it easier to specify the query parameters without having to construct a URL
+2. It returns the results as a data frame, which is easier to use within R
+
+Lets start with the query of information about Sorghum from species table from above
 
-```{r simulated-LAI}
-sorghum_lai <- betydb_query(table = 'search',
-                            trait = "LAI",
+```{r query-species}
+
+sorghum_info <- betydb_query(table = 'species',
+                            genus = "Sorghum",
                             api_version = 'beta',
-                            limit = 5000,
-                            betyurl = "https://terraref.ncsa.illinois.edu/bety-test/", 
-                            key = readLines('~/.betykey_public', warn = FALSE))
+                            limit = 'none',
+                            betyurl = "https://terraref.ncsa.illinois.edu/bety/", 
+                            key = readLines('~/.betykey', warn = FALSE))
 
 ```
 
-Notice all of the arguments? We can change this by setting the default options 
+#### R - setting options for the traits package
+
+Notice all of the arguments that the `betydb_query` function requires? We can change this by setting the default connection options thus:
 
 
 ```{r}
-options(betydb_key = readLines('~/.betykey_public', warn = FALSE),
-        betydb_url = "https://terraref.ncsa.illinois.edu/bety-test/",
+options(betydb_key = readLines('~/.betykey', warn = FALSE),
+        betydb_url = "https://terraref.ncsa.illinois.edu/bety/",
         betydb_api_version = 'beta')
 ```
 
 Now the same query can be reduced to:
 
-```{r eval=FALSE}
-sorghum_lai <- betydb_query(table = 'search',
-                            trait = "LAI",
-                            limit = 5000)
+```{r sv_area}
+sorghum_height <- betydb_query(table = 'search',
+                               trait = "plant_height",
+                               site  = "~MAC",
+                               api_version = 'beta',
+                               limit = 'none',
+                               betyurl = "https://terraref.ncsa.illinois.edu/bety/", 
+                               key = readLines('~/.betykey', warn = FALSE))
 ```
 
+### Time series of height
+
+Now we can take a look at the data that we have just queried. 
 
 ```{r}
-ggplot(data = sorghum_lai) +
-  geom_smooth(aes(x = lubridate::yday(lubridate::ymd_hms(raw_date)), y = mean, color = as.factor(lubridate::year(lubridate::ymd_hms(raw_date)))), span = 0.5) +
+ggplot(data = sorghum_height, 
+       aes(x = lubridate::yday(lubridate::ymd_hms(raw_date)), y = mean, color = cultivar)) +
+  geom_smooth(se = FALSE, size = 0.5) +
+  geom_point(size = 0.5, position = position_jitter(width = 0.1)) + 
 #  scale_x_datetime(date_breaks = '6 months', date_labels = "%b %Y") +
-  ylim(c(0,6)) + 
-  ylab("Day of Year") + xlab("Leaf Area Index") +  
-  labs(color='Year')
+#  ylim(c(0,6)) + 
+  xlab("Day of Year") + ylab("Plant Height") + 
+  guides(color = guide_legend(title = 'Genotype')) +
+  theme_bw()
 
 ```