Skip to content

Commit e077662

Browse files
ifentyifenty
authored
Merge pull request #79 from ECCO-GROUP/ian_20240509
Reworking of ec2 setup routines
2 parents bb38954 + b4064c5 commit e077662

8 files changed

Lines changed: 364 additions & 146 deletions

‎Cloud_Setup/JPL_setup_instructions.md‎

100644100755
Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# JPL setup for AWS EC2 instances
22

3+
(valid as of 2024-05-10)
4+
35
If you are based at JPL and setting up an AWS EC2 instance, there are some steps you need to take to successfully set up the instance to comply with JPL's security requirements and enable `ssh` access on your instance (which is no longer enabled by default). Please follow these steps in place of Steps 2 and 3 of the [AWS Cloud: getting started](https://ecco-v4-python-tutorial.readthedocs.io/AWS_Cloud_getting_started.html) for general users.
46

57
## Step 2: Start a JPL EC2 instance
@@ -35,21 +37,26 @@ JPL does not enable `ssh` access to AWS instances by default, instead preferring
3537
- *Initial set up and download GitHub repository*: Copy the following commands and paste in your SSM window (using shift-insert or right-click then **Paste**):
3638

3739
```
38-
cd ~ && sudo dnf update -y && sudo dnf install git -y && git clone https://github.com/ECCO-GROUP/ECCO-v4-Python-Tutorial.git
40+
$ cd ~ && sudo dnf update -y && sudo dnf install git -y
41+
$ sudo su jpluser
42+
$ cd ~/
43+
$ mkdir git_repos
44+
$ cd git_repos
45+
$ git clone https://github.com/ECCO-GROUP/ECCO-v4-Python-Tutorial.git
3946
```
4047

4148
- *Enable ssh access*: There is a script in the GitHub repository to enable ssh access `sshd_enable.sh`. You want to run it as the *root* user, otherwise you will not have the necessary permissions. Again, copy and paste the following in your SSM window:
4249

4350
```
44-
sudo ~/ECCO-v4-Python-Tutorial/Cloud_Setup/sshd_enable.sh
51+
$ sudo ~/git_repos/ECCO-v4-Python-Tutorial/Cloud_Setup/sshd_enable.sh
4552
```
4653

4754
The script will ask if you want to move the git repo and change its ownership. Answer **Y** and enter **jpluser** for user name.
4855

4956
Once the script is completed, you should be able to ssh into your new instance. You can **Terminate** the SSM window. Then from your machine's terminal window, connect to the instance's *private* IPv4 address (given on the AWS instance summary page) with user name **jpluser**. For example, if the key file is `~/.ssh/aws_ec2_jupyter.pem` and the private IPv4 address is 100.104.70.37, then:
5057

5158
```
52-
ssh -i "~/.ssh/aws_ec2_jupyter.pem" jpluser@100.104.70.37 -L 9889:localhost:9889
59+
$ ssh -i "~/.ssh/aws_ec2_jupyter.pem" jpluser@100.104.70.37 -L 9889:localhost:9889
5360
```
5461

5562
The `-L` option indicates a tunnel from the local machine's port 9889 to the instance's port 9889; this will be used later to open Jupyterlab through your local machine's web browser.
@@ -62,7 +69,7 @@ Now you need to install software (conda/miniconda/miniforge) to run Python, and
6269

6370
1. Installing `tmux` (which allows us to persist tasks on a remote machine even when disconnected).
6471

65-
1. Downloading `Miniforge.sh` from *conda-forge* which enables us to install `conda` and `mamba` (a faster, C-based `conda`) in the `/tmp` directory.
72+
1. Downloading `Miniforge.sh` from *conda-forge* which enables us to install `conda` and `mamba` (a faster, C-based `conda`)
6673

6774
1. Creating a new conda environment called `jupyter` that will contain the packages we need to run the notebooks.
6875

@@ -73,7 +80,7 @@ Now you need to install software (conda/miniconda/miniforge) to run Python, and
7380
To run `jupyter_env_setup.sh`, copy, paste, and execute the following two commands on the instance:
7481

7582
```
76-
sudo chmod 755 ~/ECCO-v4-Python-Tutorial/Cloud_Setup/jupyter_env_setup.sh && ~/ECCO-v4-Python-Tutorial/Cloud_Setup/jupyter_env_setup.sh
83+
$ sudo chmod 755 ~/git_repos/ECCO-v4-Python-Tutorial/Cloud_Setup/jupyter_env_setup.sh && ~/git_repos/ECCO-v4-Python-Tutorial/Cloud_Setup/jupyter_env_setup.sh
7784
```
7885

79-
The script takes a few minutes to run, but it should set up our environment with the packages we need. Now you can return to Step 4 of the [AWS Cloud: getting started](https://ecco-v4-python-tutorial.readthedocs.io/AWS_Cloud_getting_started.html) tutorial.
86+
The script takes several minutes to run, but it should set up our environment with the packages we need. Now you can return to Step 4 of the [AWS Cloud: getting started](https://ecco-v4-python-tutorial.readthedocs.io/AWS_Cloud_getting_started.html) tutorial.

‎Cloud_Setup/install_conda_mamba.sh‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
#!/bin/bash
2+
3+
# Shell script fgr setting up Miniforge/conda, jupyter, essential Python packages on an AWS EC2 instance.
4+
# Files will go into the ~/conda/ directory
5+
6+
# # Start body of script
7+
8+
red_start='\033[0;31m'
9+
blue_start='\033[0;34m'
10+
nocolor_start='\033[0m'
11+
12+
13+
# only proceed if both tmux and wget are installed
14+
15+
if ! command -v wget &> /dev/null; then
16+
echo "wget not installed, install first then re-run this script"
17+
exit 1
18+
elif ! command -v tmux &> /dev/null; then
19+
echo "tmux not installed, install first then re-run this script"
20+
exit 1
21+
fi
22+
23+
24+
# retrieve and install miniforge
25+
echo -e "${red_start}Starting Miniforge3 installation${nocolor_start}"
26+
27+
# download Miniforge
28+
if [ -f ~/Miniforge3.sh ]; then
29+
wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh" -O ~/Miniforge3.sh
30+
else
31+
echo "Miniforge already downloaded!"
32+
fi
33+
34+
# install Miniforge
35+
# default directory is ~/miniforge3
36+
37+
sh ~/Miniforge3.sh -b
38+
39+
# add conda and mamba to .bashrc or equivalent
40+
~/miniforge3/bin/conda init
41+
~/miniforge3/bin/mamba init
42+
43+
echo -e "${red_start}Completed Miniforge3 installation${nocolor_start}"
44+
echo -e "${red_start}Restart your shell${nocolor_start}"
45+
46+
mamba update -n base -c conda-forge conda
47+
48+
# this one is a bit of a mystery, but required
49+
echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/miniforge3/lib" >> ~/.bashrc

‎Cloud_Setup/install_packages.sh‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
#!/bin/bash
2+
3+
# Shell script for setting up conda, jupyter, essential Python packages on an AWS EC2 instance.
4+
# Assumes that the ECCO-v4-Python-Tutorial Github repository has already been downloaded using:
5+
#
6+
# $ sudo dnf update -y
7+
# $ sudo dnf install git -y
8+
# $ cd ~
9+
# $ git clone https://github.com/ECCO-GROUP/ECCO-v4-Python-Tutorial.git
10+
11+
# Then run this script:
12+
#
13+
# $ sudo chmod 755 ~/ECCO-v4-Python-Tutorial/Cloud_Setup/jupyter_env_setup.sh
14+
# $ ~/ECCO-v4-Python-Tutorial/Cloud_Setup/jupyter_env_setup.sh
15+
16+
17+
18+
# # Start body of script
19+
20+
red_start='\033[0;31m'
21+
blue_start='\033[0;34m'
22+
nocolor_start='\033[0m'
23+
24+
# # set paths to environment and package directories
25+
26+
# create jupyter environment
27+
mamba create --name jupyter python=3.11 -y
28+
echo -e "${red_start}Created jupyter environment${nocolor_start}"
29+
30+
# install python packages (using mamba) in jupyter environment
31+
mamba activate jupyter
32+
echo -e "${red_start}Installing Python packages in jupyter environment${nocolor_start}"
33+
mamba install requests tqdm numpy pandas -y
34+
mamba install xorg-libice libexpat libevent -y
35+
mamba install nspr alsa-lib libogg libpq -y
36+
mamba install xorg-renderproto xorg-xf86vidmodeproto graphite2 expat -y
37+
mamba install libgpg-error dbus -y
38+
mamba install libflac gettext -y
39+
mamba install xcb-util-wm xorg-libx11 xcb-util-image -y
40+
mamba install xkeyboard-config -y
41+
mamba install libxkbcommon fonts-conda-forge font-ttf-ubuntu gstreamer zlib -y
42+
mamba install xorg-xextproto libpng attr mpg123 -y
43+
mamba install pixman libvorbis glib-tools -y
44+
mamba install libsystemd0 xcb-util-keysyms xorg-libxrender libllvm15 -y
45+
mamba install font-ttf-dejavu-sans-mono pcre2 font-ttf-inconsolata font-ttf-source-code-pro -y
46+
mamba install lame nss xorg-xproto pthread-stubs xorg-libxdmcp -y
47+
mamba install libgcrypt xorg-libsm xorg-libxext fonts-conda-ecosystem xorg-kbproto mysql-libs -y
48+
mamba install fontconfig libjpeg-turbo xcb-util-renderutil -y
49+
mamba install glib -y
50+
mamba install freetype libcap libcups libopus -y
51+
mamba install gst-plugins-base mysql-common xcb-util -y
52+
mamba install cairo -y
53+
mamba install libsndfile harfbuzz xorg-libxau -y
54+
mamba install libglib libxcb -y
55+
mamba install qt-main -y
56+
mamba install pyqt -y
57+
mamba install matplotlib -y
58+
mamba install netcdf4 -y
59+
mamba install h5netcdf -y
60+
mamba install boto3 lxml -y
61+
mamba install scipy -y
62+
mamba install geos -y
63+
mamba install proj pyproj -y
64+
mamba install cartopy -y
65+
mamba install notebook -y
66+
mamba install progressbar -y
67+
mamba install gsw -y
68+
mamba install nco -y
69+
mamba install pympler -y
70+
71+
# install remaining packages using pip
72+
# (mamba installs tend to get killed on t2.micro)
73+
pip install dask
74+
pip install "xarray[complete]"
75+
pip install jupyterlab
76+
pip install dask_labextension
77+
pip install s3fs
78+
pip install ecco_v4_py
79+
80+
echo -e "${red_start}Completed Python package installations${nocolor_start}"
81+
82+
83+
echo -e "${red_start}Setting up NASA Earthdata authentication${nocolor_start}"
84+
# NASA Earthdata authentication
85+
# check if credentials are already archived in ~/.netrc, and if not then prompt the user for them
86+
earthdata_cred_stored=0
87+
if [ -f ~/.netrc ]; then
88+
if grep -q "machine urs.earthdata.nasa.gov" ~/.netrc; then
89+
earthdata_cred_stored=1
90+
echo -e "${red_start}Earthdata credentials already archived ${nocolor_start}"
91+
fi
92+
fi
93+
if [ $earthdata_cred_stored -eq 0 ]; then
94+
if [ -f ~/.netrc ]; then sudo chmod 600 ~/.netrc; fi
95+
read -p 'NASA Earthdata username: ' uservar
96+
read -sp 'NASA Earthdata password: ' passvar
97+
echo -e "machine urs.earthdata.nasa.gov\n login ${uservar}\n password ${passvar}\n" >> ~/.netrc
98+
99+
echo -e "\n${red_start}NASA Earthdata authentication info archived in ~/.netrc${nocolor_start}"
100+
fi
101+
sudo chmod 400 ~/.netrc
102+
103+
# create symlink to jupyter_lab_start.sh from the user's home directory
104+
ln -s ~/ECCO-v4-Python-Tutorial/Cloud_Setup/jupyter_lab_start.sh ~/jupyter_lab_start.sh
105+
106+
echo "goodbye!"

‎Cloud_Setup/install_packages_bigmem.sh‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
#!/bin/bash
2+
3+
# Shell script for setting up conda, jupyter, essential Python packages on an AWS EC2 instance.
4+
5+
# before installing, wise to update the local
6+
# machine with the latest packages.
7+
# on redhat linux machines,
8+
# $ sudo dnf update -y
9+
# $ sudo dnf install git -y
10+
11+
# # Start body of script
12+
13+
red_start='\033[0;31m'
14+
blue_start='\033[0;34m'
15+
nocolor_start='\033[0m'
16+
17+
# # set paths to environment and package directories
18+
# create conda environment called 'jupyter'
19+
env_name='jupyter'
20+
21+
conda create --name ${env_name} python=3.12 -y
22+
echo -e "${red_start}Created ${env_name} conda environment${nocolor_start}"
23+
24+
# install python packages (using mamba) in jupyter environment
25+
conda activate ${env_name}
26+
27+
echo -e "${red_start}Installing Python packages in jupyter environment${nocolor_start}"
28+
29+
# -n syntax installs the packages into the specified environment name
30+
mamba install -n ${env_name} requests tqdm numpy pandas xorg-libice libexpat libevent -y
31+
mamba install -n ${env_name} nspr alsa-lib libogg libpq xorg-renderproto xorg-xf86vidmodeproto graphite2 expat -y
32+
mamba install -n ${env_name} libgpg-error dbus libflac gettext xcb-util-wm xorg-libx11 xcb-util-image -y
33+
mamba install -n ${env_name} xkeyboard-config libxkbcommon fonts-conda-forge font-ttf-ubuntu gstreamer zlib -y
34+
mamba install -n ${env_name} xorg-xextproto libpng attr mpg123 pixman libvorbis glib-tools -y
35+
mamba install -n ${env_name} libsystemd0 xcb-util-keysyms xorg-libxrender libllvm15 -y
36+
mamba install -n ${env_name} font-ttf-dejavu-sans-mono pcre2 font-ttf-inconsolata font-ttf-source-code-pro -y
37+
mamba install -n ${env_name} lame nss xorg-xproto pthread-stubs xorg-libxdmcp -y
38+
mamba install -n ${env_name} libgcrypt xorg-libsm xorg-libxext fonts-conda-ecosystem xorg-kbproto mysql-libs -y
39+
mamba install -n ${env_name} fontconfig libjpeg-turbo xcb-util-renderutil -y
40+
mamba install -n ${env_name} glib freetype libcap libcups libopus -y
41+
mamba install -n ${env_name} gst-plugins-base mysql-common xcb-util -y
42+
mamba install -n ${env_name} cairo libsndfile harfbuzz xorg-libxau -y
43+
mamba install -n ${env_name} libglib libxcb qt-main pyqt matplotlib netcdf4 -y
44+
mamba install -n ${env_name} h5netcdf boto3 lxml scipy goes proj pyproj cartopy notebook -y
45+
mamba install -n ${env_name} progressbar gsw nco pympler -y
46+
mamba install -n ${env_name} xarray[complete] jupyterlab dask_labextension s3fs -y
47+
mamba install -n ${env_name} pyresample -y
48+
49+
# install remaining packages using pip
50+
# (mamba installs tend to get killed on t2.micro)
51+
pip install ecco_v4_py --no-cache-dir
52+
53+
echo -e "${red_start}Completed Python package installations${nocolor_start}"
54+
55+
# Set up NASA Earthdata credential
56+
57+
echo -e "${red_start}Setting up NASA Earthdata authentication${nocolor_start}"
58+
# NASA Earthdata authentication
59+
# check if credentials are already archived in ~/.netrc, and if not then prompt the user for them
60+
earthdata_cred_stored=0
61+
if [ -f ~/.netrc ]; then
62+
if grep -q "machine urs.earthdata.nasa.gov" ~/.netrc; then
63+
earthdata_cred_stored=1
64+
echo -e "${red_start}Earthdata credentials already archived ${nocolor_start}"
65+
fi
66+
fi
67+
if [ $earthdata_cred_stored -eq 0 ]; then
68+
if [ -f ~/.netrc ]; then sudo chmod 600 ~/.netrc; fi
69+
read -p 'NASA Earthdata username: ' uservar
70+
read -sp 'NASA Earthdata password: ' passvar
71+
echo -e "machine urs.earthdata.nasa.gov\n login ${uservar}\n password ${passvar}\n" >> ~/.netrc
72+
73+
echo -e "\n${red_start}NASA Earthdata authentication info archived in ~/.netrc${nocolor_start}"
74+
fi
75+
sudo chmod 400 ~/.netrc
76+
77+
# create symlink to jupyter_lab_start.sh from the user's home directory
78+
ln -s ~/ECCO-v4-Python-Tutorial/Cloud_Setup/jupyter_lab_start.sh ~/jupyter_lab_start.sh
79+
80+
echo "goodbye!"

‎Cloud_Setup/install_wget_tmux.sh‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
#!/bin/bash
2+
3+
# script to install wget and tmux
4+
# wget is needed to download Miniforge
5+
# tmux is useful (but not really necessary) for running jupyter
6+
7+
# # Start body of script
8+
9+
red_start='\033[0;31m'
10+
blue_start='\033[0;34m'
11+
nocolor_start='\033[0m'
12+
13+
if command -v dnf &> /dev/null
14+
then
15+
# install wget and tmux
16+
sudo dnf install wget tmux -y
17+
elif command -v apt &> /dev/null; then
18+
sudo apt install wget tmux -y
19+
elif command -v yum &> /dev/null; then
20+
sudo yum install wget tmux -y
21+
fi
22+
23+
if command -v wget &> /dev/null; then
24+
echo -e "${red_start}Installed wget${nocolor_start}"
25+
else
26+
echo -e "${blue_start}Could not install wget! -- install it yourself ${nocolor_start}"
27+
fi
28+
29+
if command -v tmux &> /dev/null; then
30+
echo -e "${red_start}Installed tmux${nocolor_start}"
31+
else
32+
echo -e "${blue_start}Could not install tmux! -- install it yourself ${nocolor_start}"
33+
fi
34+

0 commit comments

Comments
 (0)