You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+23-9Lines changed: 23 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,20 +1,34 @@
1
1
# Tutorials
2
2
3
-
Learn to use TERRA REF data and software
3
+
## An introduction to the use of TERRA REF data and software
4
4
5
-
Note that many of these tutorials have complex dependencies. These can be launched from within the [TERRA REF Sensor Data Portal](https://terraref.ncsa.illinois.edu) (requires account / access).
5
+
Many of these tutorials have complex dependencies.
6
6
7
-
Also try:
7
+
### Data Access
8
8
9
-
* mybinder.org/repo/terraref/rstudio (for R tutorials under traits/)
10
-
* mybinder.org/repo/terraref/rstudio-geospatial (for R tutorials under sensors/)
* mybinder.org/repo/terraref/jupyter-netcdf (Python tutorials under sensors/)
9
+
The first research-grade version of TERRA REF data products will be released in November 2018.
10
+
Before that, we will make evaluation releases available: the alpha version was released in November 2016 and the beta version will be released in 2017.
13
11
12
+
make the data available for evaluation with the goal of receiving feedback.
13
+
are making
14
+
Many access data that is available online, though most require authentication.
15
+
16
+
Some make use of very large files that are available.
17
+
These can be launched from within the [TERRA REF Sensor Data Portal](https://terraref.ncsa.illinois.edu) (requires account / access).
18
+
19
+
###
20
+
21
+
22
+
### Links
23
+
24
+
TODO: add links to quick-start documentation, README's, code for learning and applied examples
25
+
26
+
* Data portal: terraref.org/data
27
+
* Docker Images on Docker Hub: hub.docker.com/terraref
28
+
*
29
+
### References
14
30
15
-
## Links
16
31
17
-
to quick-start documentation, README's, code for learning and applied examples
Copy file name to clipboardExpand all lines: traits/00-BETYdb-getting-started.Rmd
+115-7Lines changed: 115 additions & 7 deletions
Original file line number
Diff line number
Diff line change
@@ -1,16 +1,124 @@
1
+
---
2
+
title: "Getting Started with BETYdb"
3
+
author: "David LeBauer"
4
+
date: "`r Sys.Date()`"
5
+
output: html_document
6
+
---
1
7
2
-
## Sign up
8
+
```{r setup}
9
+
library(traits)
10
+
knitr::opts_chunk$set(echo = FALSE, cache = TRUE)
11
+
library(ggplot2)
12
+
library(ggthemes)
13
+
library(GGally)
14
+
theme_set(theme_bw())
15
+
library(dplyr)
16
+
```
3
17
4
-
## Download from web interface
5
18
19
+
## TERRA Ref Trait Database
6
20
21
+
The TERRA Ref program uses the BETYdb database and web application software to store plant and plot level trait data.
7
22
8
-
##API Summary
23
+
### BETYdb: database software and web application
9
24
10
-
API key sent in the mail
25
+
The BETYdb software is actively used and developed by the [TERRA Reference](terraref.org) program as well as by the [PEcAn project](pecanproject.org).
11
26
12
-
Save it to textfile (take from traits/02- file
27
+
For more information about BETYdb, see the following:
28
+
29
+
* BETYdb documentation (available via the web application under 'Docs')
30
+
*_Data Access_
31
+
*_Data Entry Workflow:_ how to add data to the database
32
+
*_BETYdb Technical Documentation_ is written for advanced users and website and database administrators who may also be interested in the [full database schema](betydb.org/schemas)
33
+
* BETYdb: A Yield, Trait and Ecosystem Service Database Applied to Second Generation Bioenergy Feedstocks. ([LeBauer et al, 2017](dx.doi.org/10.1111/gcbb.12420))
34
+
35
+
The TERRA REF trait database (terraref.ncsa.illinois.edu/bety) uses the BETYdb data schema (structure) and web application.
36
+
There are at least a half-dozen other databases using the BETYdb software that these exercises will work with, though the results will depend on the available data.
37
+
The first, betydb.org is described in LeBauer et al, 2017.
38
+
Others are listed in the 'distributed BETYdb' section of the technical documentation.
39
+
40
+
One database, terraref.ncsa.illinois.edu/terra-test, houses a simulated dataset that is used in [lesson 1: A simulated data set](../traits/01-simulated-sorghum.Rmd) and does not require an account to access the data.
41
+
BETYdb is only designed to keep the primary data private. Metadata such as field management and experimental design are available if the url is public.
42
+
43
+
## Getting an account for the TERRA trait database
44
+
45
+
* sign up for an account at terraref.ncsa.illinois.edu/bety
46
+
* sign up for alpha user [link to form]
47
+
* wait for database access to be granted
48
+
* Your API key will be sent in the email; it can also be found - and regenerated - by navigating to 'data --> users' in the web interface
49
+
50
+
TODO add signup info from handout
51
+
52
+
## First steps: download data from web interface
53
+
54
+
TODO add steps to download csv from the web interface
55
+
56
+
Note that the web interface only provides a core set of data. More complex queries, such as those in the [Agronomic metadata](../traits/04-agronomic-metadata.Rmd)
57
+
58
+
## Advanced: Using URLs to construct Queries
59
+
60
+
The first step toward reproducible pipelines is to automate the process of searching the database and returning results. This is one of the key roles of an Application programming interface, or 'API'. You can learn to use the API in less than 20 minutes, starting now.
61
+
62
+
### What is an API?
63
+
64
+
An API is ...
65
+
66
+
### Using Your API key to Connect
67
+
68
+
An API key is like a password. It allows you to access data, and should be kept private.
69
+
Therefore, we are not going to put it in code that we share. The one exception is the key 9999999999999999999999999999999999999999 that will allow you to access metadata tables (all tables except _traits_ and _yields_).
70
+
71
+
A common way of handling private API keys is to place it in a text file in your home directory.
72
+
Don't put it in a project directory where it might be inadvertently shared.
73
+
74
+
Here is how to find and save your API key:
75
+
76
+
* click file --> new --> text file
77
+
* copy the api key that was sent when you registered into the file
78
+
* file --> save as '~/.betykey'
79
+
80
+
Equivalently in R `r writeLines('9999999999999999999999999999999999999999', con = '~/.betykey')` or at the command line `sh echo 9999999999999999999999999999999999999999 > ~/.betykey`
81
+
82
+
For the purposes of the tutorial, you can assign it to the `betykey` variable in the console window.
83
+
84
+
### Constructing a URL query
85
+
86
+
First, lets construct a query by putting together a URL.
87
+
88
+
1. start with the database url: `terraref.ncsa.illinois.edu/bety`
89
+
* this url brings you to the home page
90
+
2. Add the path to the API, `/api/beta`
91
+
* now we have terraref.ncsa.illinois.edu/bety/api/beta, which points to the API documentation
92
+
3. Add the name of the table you want to query. Lets start with `variables`
Copy file name to clipboardExpand all lines: traits/01-simulated-sorghum.Rmd
+19-28Lines changed: 19 additions & 28 deletions
Original file line number
Diff line number
Diff line change
@@ -6,6 +6,7 @@ output: html_document
6
6
---
7
7
8
8
```{r setup}
9
+
library(traits)
9
10
knitr::opts_chunk$set(echo = FALSE, cache = TRUE)
10
11
library(ggplot2)
11
12
library(ggthemes)
@@ -14,9 +15,14 @@ theme_set(theme_bw())
14
15
library(dplyr)
15
16
```
16
17
17
-
# Background: The design of a simulated dataset
18
+
# Working with a simulated dataset?
19
+
20
+
To explore the potential of phenotyping data, we have simulated the type of data that might be observed by daily scans by drone or robot. These data are freely accessible, and thus useful for teaching and exploration that does not require access.
21
+
22
+
## Methods:
23
+
24
+
### The design of a simulated dataset
18
25
19
-
To explore the potential of phenotyping data, we have simulated the type of data that might be observed by daily scans by drone or robot.
20
26
21
27
We have simulated 500 genotypes across 12 sites and five years using a mechanistic model.
22
28
@@ -26,19 +32,19 @@ All of these simulated datasets are released with an unrestrive [copyright](http
26
32
27
33
### A note on variable names
28
34
29
-
I have used the variable names currently used in BETYdb.org/variables, along with names inspired by the more standardized naming Climate Forecasting conventions. However, this is a very early pre-release, and we welcome comments on how such data should be formatted and accessed can be discussed on GitHub.
35
+
The variable names are those currently used in BETYdb.org/variables, along with names inspired by the more standardized naming Climate Forecasting conventions. However, this is a very early pre-release, and we welcome comments on how such data should be formatted and accessed can be discussed on GitHub.
30
36
31
-
> Exercise: can you locate the relevant issues?
37
+
> Exercise: can you locate the GitHub issues that discuss trait data standards and variable naming conventions?
32
38
33
39
[This is a slideshow](https://docs.google.com/presentation/d/10aN_5whs8y9SOC8Y9Rj1kbWCfi7YG3yFyNqVX4JUr2U/edit?usp=sharing) of interfaces from the broader community that could serve as a common interface.
34
40
35
-
# Design of Simulation Experiment
41
+
###Design of Simulation Experiment
36
42
37
43
500 Sorghum lines grown at each of three sites, four blocks per site, along a N-S transect in Illinois over five years (2021-2025).
38
44
39
-
## Time Span (2021-2025)
45
+
####Time Span (2021-2025)
40
46
41
-
These are historic data, but the years have been changed to emphasize the point that these are not real data. The years have been chosen to select climate extremes. Two years were dry, two were wet, and one was average.
47
+
While the climate data are derived from historic data, the years have been changed to reinforce the fact that these are not real data. The years have been chosen to select climate extremes. Two years were dry, two were wet, and one was average.
42
48
43
49
| year | drought index |
44
50
|-----|-----|
@@ -71,11 +77,11 @@ Each site has four replicate fields: A, B, C, D. This simulated dataset assumes
Two-hundred and twenty-seven lines were grown at each site. Each line is identified by a unique integer in the range [9915:10141]
77
83
78
-
## Phenotypes
84
+
###Phenotypes
79
85
80
86
The phenotypes associated with each genotype is in the file `phenotypes.csv`.
81
87
@@ -117,36 +123,21 @@ This dataset includes what a sensor might observe, daily for five years during t
117
123
|| Height | canopy_height | m ||
118
124
119
125
120
-
# Quick start
121
-
122
-
123
-
For simplicity, and because I neither have a model nor the data to simulate _Sorghum_, I have started with some phenotypes and simulations of plant growth based on a model that simulates the growth of biomass crops including Miscanthus, Switchgrass, Sugarcane, and coppice Willow.
124
126
125
-
## Some background / methods.
126
-
127
-
I start with simulation of Miscanthus, over Illinois as a proxy for Sorghum, since I have a model, [BioCro, Miguez et al, 2009](github.com/ebimodeling/biocro), that simulates Miscanthus. [The code used to run the model and add noise is on GitHub](https://github.com/ebimodeling/biocro_regional/edit/master/vignettes/regional_pecan_workflow.Rmd).
128
-
Like Sorghum, Miscanthus uses C4 photosynthesis, and this is used to compute carbon uptake at hourly time steps in the simulation model.
129
-
Unlike Sorghum, Miscanthus grows clonally and is propagated by Rhizome instead of by seed. Furthermore, Miscanthus is perennial: it re-grows each year from carbon stored in rhizomes.
127
+
I start with simulation over Illinois using another grass with C4 photosynthesis, [BioCro, Miguez et al, 2009](github.com/ebimodeling/biocro). [The code used to run the model and add noise is on GitHub](https://github.com/ebimodeling/biocro_regional/edit/master/vignettes/regional_pecan_workflow.Rmd).
128
+
The model computes carbon uptake at hourly time steps.
130
129
The 'genotypes' are based on five-hundred quasi-random parameterizations of a biophysical crop model.
131
130
132
-
The virtual "Miscanthus" is encoded as a set of associated species and prior estimates of the phenotypes used to parameterize the simulation model. The concept of Plant Functional Type (PFT) is any group of one or more plants, originally functionally related species such as 'C4 crops' or 'hardwood trees'. We use PFT at the finest scale of genotype. In the actual research that generated these data, this was the clone _Miscanthus x giganteus_, and this is [PFT #123 in BETYdb](https://www.betydb.org/pfts/123).
133
-
134
-
## A Simulated Sorghum Breeding Population
135
-
136
-
In make-believe land we take a bunch of data generated above and start to rename things
131
+
The virtual crop is encoded as a set of associated species and prior estimates of the phenotypes used to parameterize the simulation model.
137
132
138
-
* run_id -> genotype #represents quasi-random set of traits
139
-
* lat, lon -> To estimate "E"
133
+
* lat, lon -> To estimate "Environmental effects"
140
134
* for within site E effects, use a few points within 1/4 degree lat/lon
141
135
* for across site E effects, use southern, central, and northern Illinois.
Copy file name to clipboardExpand all lines: traits/02-danforth-phenotyping-facility.Rmd
-29Lines changed: 0 additions & 29 deletions
Original file line number
Diff line number
Diff line change
@@ -15,35 +15,6 @@ theme_set(theme_bw())
15
15
library(traits)
16
16
```
17
17
18
-
19
-
## BETYdb
20
-
21
-
BETYdb stores trait data and agronomic metadata. An introduction to the API is in ../betydb.md
22
-
23
-
Also see the full documentation for accessing data from BETYdb.
24
-
25
-
26
-
### Setting up an API key and establishing a connection
27
-
28
-
An API key is like a password. It allows you to access data, and should be kept private. Therefore, we are not going to put it in code that we share. One way to do this is to place it in a simple text file.
29
-
30
-
* click file --> new --> text file
31
-
* copy the api key that was sent when you registered into the file
32
-
* file --> save save as '/home/rstudio/.betykey'
33
-
34
-
For the purposes of the tutorial, you can assign it to the `mykey` variable in the console window.
0 commit comments