Merge branch 'traits_tutorials' of github.com:terraref/tutorials into traits_tutorials

Author: kimberlyh66

index.Rmd

@@ -6,10 +6,62 @@ date: "`r Sys.Date()`"
 documentclass: book
 output:
   bookdown::gitbook: default
-  bookdown::pdf_book: default
 ---
 
-# Preamble
+# Overview
+
+This book is intended to introduce users to TERRA REF data as quickly as possible. 
+
+It introduces to the wide range of phenomics datasets generated by the TERRA Reference program. Not only does TERRA REF have a large number of data sets, but many of the databases can be accessed in a number of different ways. While this makes it more complicated to learn, the goal is to provide users with the flexibility to access data in the most useful way.
+
+## User Accounts and permission to access TERRA REF data
+
+TODO: link to relevant parts of docs.terraref.org
+
+## Ways of Acessing Data
+
+* Web Interfaces
+* Files
+* Programming APIs
+* API Clients
+
+## Other Resources
+
+The TERRA REF website: terraref.org
+The TERRA REF Technical Documentation: docs.terraref.org
+
+## Contents
+
+Scope ...
+
+Audience ...
+
+
+## Pre-requisites
+
+While we assume that readers will have some familiarity with the nature of the problem - remote sensing of crop plants - for the most part, these tutorials assume that the user will bring their own scientific questions and a sense of curiosity and are eager to learn. 
+
+Some of the lessons only require a web browser; others will assume familarity with programming at the command line in (typically only one of) Python, R, and / or SQL. You should be willing to find help (see finding help, below).
+
+## Technical Requirements
+
+At a minimum, you should have:
+
+* An internet connection
+* Web Browser
+* A TERRA REF Beta User account
+  * If you have not done so, please sign up at terraref.org/beta
+* Access to the data that you are using
+  * The tutorials will state which databases you will need access to
+* Software:
+  * Software requirements vary with the tutorials, and may be complex
+  * 
+
+## Finding help
+
+* Slack
+* GitHub
+* Google
 
 ```{r}
 knitr::opts_chunk$set(echo = FALSE, cache = TRUE)

traits/02-betydb-api-access.Rmd

@@ -1,15 +1,27 @@
 # Accessing Trait Data Via the BETYdb API
 
-## Using URLs to construct Queries
-
-The first step toward reproducible pipelines is to automate the process of searching the database and returning results. This is one of the key roles of an Application programming interface, or 'API'. You can learn to use the API in less than 20 minutes, starting now. 
+This will teach you how to query trait data using a browser as well as using the command line tool `curl`. This interface is the primary way in which you can access data from the command line. 
 
-### What is an API?
+## What is an API?
 
 An API is an 'Application Programming Interface'. An API is a way that you and your software can connect to and access data. 
 
 All of our databases have web interfaces for humans to browse as well as APIs that are constructed as URLs. 
 
+## Tutorial Contents
+
+In this tutorial, we will describe three ways to access data using:
+
+1. A URL typed into your browser
+2. The command line, or terminal
+3. The R jsonlite package
+
+We also have interfaces using R 'traits' package or the Python 'terrautils' package that return data in a more familiar and ready to analyze tabular format; these will be described later. You can skip ahead to those chapters, but this chapter will provide some insight into the methods that underlie those libraries.
+
+## Using URLs to construct Queries
+
+The first step toward reproducible pipelines is to automate the process of searching the database and returning results. This is one of the key roles of an Application programming interface, or 'API'. You can learn to use the API in less than 20 minutes, starting now. 
+
 
 ### Using Your API key to Connect
 
@@ -27,25 +39,19 @@ Here is how to find and save your API key:
 
 For the public key, you can call this file `.betykey_public`. 
 
-### Ways to access API data
-
-1. Through a URL query
-2. Using the bash shell
-3. Using the R jsonlite package
 
+## Accessing data using a URL query
 
-### Accessing data using a URL query
 
-
-## Components of a URL query
+### Components of a URL query
 
 * base url: `terraref.ncsa.illinois.edu/bety`
 * path to the api: `/api/v1`
 * api endpoint: `/search` or `traits` or `sites`. For BETYdb, these are the names of database tables. 
 * Query parameters: `genus=Sorghum`
 * Authentication: `key=9999999999999999999999999999999999999999` is the public key for the TERRA REF traits database. 
 
-## Constructing a URL query
+### Constructing a URL query
 
 First, lets construct a query by putting together a URL.
 
@@ -72,9 +78,7 @@ First, lets construct a query by putting together a URL.
 
 What do you see? Do you think that this is all of the records? What happens if you add `&limit=none`? 
 
-
-
-#### Accessing data using the Shell
+## Accessing data using the Command Line Terminal
 
 Type the following command into a bash shell (the `-o` option names the output file): 
 
@@ -90,7 +94,7 @@ curl -o sorghum.json \
     "https://terraref.ncsa.illinois.edu/bety/api/v1/species?genus=Sorghum&key=`cat .betykey_public`"
 ```
 
-### Accessing API data using the R jsonlite package
+## Using the R jsonlite package to access the API with a URL query 
 
 ```{r text-api, warning = FALSE}
 sorghum.json <- readLines(

traits/03-access-r-traits.Rmd

@@ -1,20 +1,20 @@
 # Accessing Trait Data in R
 
-## Using the R traits package to query the database
+The rOpenSci traits package makes it easier to query the TERRA REF trait database because 1) you can pass the query parameters in an R function, and the package takes care of putting the parameters into a valid URL and 2) because the package returns data in a tabular format that is ready to analyze.
 
-The rOpenSci traits package makes it easier to query the TERRA REF trait database, or any database that uses BETYdb software.
+## Using the R traits package to query the database
 
-First, make sure we have the latest version from the terraref fork of the repository on github. (you can install using the standard `install.packages('traits')` but as of Jan 2018 this version times out on very large datasets).
+## Setup 
 
-### Install the traits package
+Install the traits package
 
-This is for users working on their own computer - if you are using the TERRA REF Rstudio Docker container (including on workbench) you can skip this step. 
+The traits package is on CRAN, and can therefore be installed using the following command:  
 
 ```{r install_traits, echo = FALSE, include = FALSE}
-devtools::install_github('terraref/traits')
+install.packages('traits')
 ```
 
-Now, we can load the packages that we will need to get started.
+Load other packages that we will need to get started.
 
 ```{r 00-setup, message = FALSE}
 library(traits)
@@ -24,7 +24,7 @@ theme_set(theme_bw())
 library(dplyr)
 ```
 
-
+Create a file that contains your API key. If you have signed up for access to the TERRA REF database, your API key will have been sent to you in an email. The public key will provide access to all metadata; you will need a personal key _and_ permissions to access the trait data. If you receive empty (NULL) datasets, it is likely that you do not have permissions. 
 
 ```{r writing-key}
 # This should be done once with the key sent to you in your email