reworking of some of the ec2 setup routines

Author: JPL User

Cloud_Setup/JPL_setup_instructions.md

@@ -35,21 +35,26 @@ JPL does not enable `ssh` access to AWS instances by default, instead preferring
 - *Initial set up and download GitHub repository*: Copy the following commands and paste in your SSM window (using shift-insert or right-click then **Paste**):
 
 ```
-cd ~ && sudo dnf update -y && sudo dnf install git -y && git clone https://github.com/ECCO-GROUP/ECCO-v4-Python-Tutorial.git
+$ cd ~ && sudo dnf update -y && sudo dnf install git -y
+$ sudo su jpluser
+$ cd ~/
+$ mkdir git_repos
+$ cd git_repos
+$ git clone https://github.com/ECCO-GROUP/ECCO-v4-Python-Tutorial.git
 ```
 
 - *Enable ssh access*: There is a script in the GitHub repository to enable ssh access `sshd_enable.sh`. You want to run it as the *root* user, otherwise you will not have the necessary permissions. Again, copy and paste the following in your SSM window:
 
 ```
-sudo ~/ECCO-v4-Python-Tutorial/Cloud_Setup/sshd_enable.sh
+$ sudo ~/git_repos/ECCO-v4-Python-Tutorial/Cloud_Setup/sshd_enable.sh
 ```
 
 The script will ask if you want to move the git repo and change its ownership. Answer **Y** and enter **jpluser** for user name.
 
 Once the script is completed, you should be able to ssh into your new instance. You can **Terminate** the SSM window. Then from your machine's terminal window, connect to the instance's *private* IPv4 address (given on the AWS instance summary page) with user name **jpluser**. For example, if the key file is `~/.ssh/aws_ec2_jupyter.pem` and the private IPv4 address is 100.104.70.37, then:
 
 ```
-ssh -i "~/.ssh/aws_ec2_jupyter.pem" jpluser@100.104.70.37 -L 9889:localhost:9889
+$ ssh -i "~/.ssh/aws_ec2_jupyter.pem" jpluser@100.104.70.37 -L 9889:localhost:9889
 ```
 
 The `-L` option indicates a tunnel from the local machine's port 9889 to the instance's port 9889; this will be used later to open Jupyterlab through your local machine's web browser.
@@ -62,7 +67,7 @@ Now you need to install software (conda/miniconda/miniforge) to run Python, and
 
 1. Installing `tmux` (which allows us to persist tasks on a remote machine even when disconnected).
 
-1. Downloading `Miniforge.sh` from *conda-forge* which enables us to install `conda` and `mamba` (a faster, C-based `conda`) in the `/tmp` directory.
+1. Downloading `Miniforge.sh` from *conda-forge* which enables us to install `conda` and `mamba` (a faster, C-based `conda`)
 
 1. Creating a new conda environment called `jupyter` that will contain the packages we need to run the notebooks.
 
@@ -73,7 +78,7 @@ Now you need to install software (conda/miniconda/miniforge) to run Python, and
 To run `jupyter_env_setup.sh`, copy, paste, and execute the following two commands on the instance:
 
 ```
-sudo chmod 755 ~/ECCO-v4-Python-Tutorial/Cloud_Setup/jupyter_env_setup.sh && ~/ECCO-v4-Python-Tutorial/Cloud_Setup/jupyter_env_setup.sh
+$ sudo chmod 755 ~/git_repos/ECCO-v4-Python-Tutorial/Cloud_Setup/jupyter_env_setup.sh && ~/git_repos/ECCO-v4-Python-Tutorial/Cloud_Setup/jupyter_env_setup.sh
 ```
 
-The script takes a few minutes to run, but it should set up our environment with the packages we need. Now you can return to Step 4 of the [AWS Cloud: getting started](https://ecco-v4-python-tutorial.readthedocs.io/AWS_Cloud_getting_started.html) tutorial.
+The script takes several minutes to run, but it should set up our environment with the packages we need. Now you can return to Step 4 of the [AWS Cloud: getting started](https://ecco-v4-python-tutorial.readthedocs.io/AWS_Cloud_getting_started.html) tutorial.

Cloud_Setup/install_conda_mamba.sh

@@ -0,0 +1,49 @@
+#!/bin/bash
+
+# Shell script fgr setting up Miniforge/conda, jupyter, essential Python packages on an AWS EC2 instance.
+# Files will go into the ~/conda/ directory 
+
+# # Start body of script
+
+red_start='\033[0;31m'
+blue_start='\033[0;34m'
+nocolor_start='\033[0m'
+
+
+# only proceed if both tmux and wget are installed
+
+if ! command -v wget &> /dev/null; then
+   echo "wget not installed, install first then re-run this script"
+   exit 1
+elif ! command -v tmux &> /dev/null; then
+   echo "tmux not installed, install first then re-run this script"
+   exit 1
+fi
+
+
+# retrieve and install miniforge
+echo -e "${red_start}Starting Miniforge3 installation${nocolor_start}"
+
+# download Miniforge 
+if [ -f ~/Miniforge3.sh ]; then
+   wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh" -O ~/Miniforge3.sh
+else
+   echo "Miniforge already downloaded!"
+fi
+
+# install Miniforge 
+# default directory is ~/miniforge3
+
+sh ~/Miniforge3.sh -b
+
+# add conda and mamba to .bashrc or equivalent
+~/miniforge3/bin/conda init
+~/miniforge3/bin/mamba init
+
+echo -e "${red_start}Completed Miniforge3 installation${nocolor_start}"
+echo -e "${red_start}Restart your shell${nocolor_start}"
+
+mamba update -n base -c conda-forge conda
+
+# this one is a bit of a mystery, but required
+echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/miniforge3/lib" >> ~/.bashrc

Cloud_Setup/install_packages.sh

@@ -0,0 +1,106 @@
+#!/bin/bash
+
+# Shell script for setting up conda, jupyter, essential Python packages on an AWS EC2 instance.
+# Assumes that the ECCO-v4-Python-Tutorial Github repository has already been downloaded using:
+# 
+# $ sudo dnf update -y
+# $ sudo dnf install git -y
+# $ cd ~
+# $ git clone https://github.com/ECCO-GROUP/ECCO-v4-Python-Tutorial.git
+
+# Then run this script:
+# 
+# $ sudo chmod 755 ~/ECCO-v4-Python-Tutorial/Cloud_Setup/jupyter_env_setup.sh
+# $ ~/ECCO-v4-Python-Tutorial/Cloud_Setup/jupyter_env_setup.sh
+
+
+
+# # Start body of script
+
+red_start='\033[0;31m'
+blue_start='\033[0;34m'
+nocolor_start='\033[0m'
+
+# # set paths to environment and package directories
+
+# create jupyter environment
+mamba create --name jupyter python=3.11 -y
+echo -e "${red_start}Created jupyter environment${nocolor_start}"
+
+# install python packages (using mamba) in jupyter environment
+mamba activate jupyter
+echo -e "${red_start}Installing Python packages in jupyter environment${nocolor_start}"
+mamba install requests tqdm numpy pandas -y
+mamba install xorg-libice libexpat libevent -y
+mamba install nspr alsa-lib libogg libpq -y
+mamba install xorg-renderproto xorg-xf86vidmodeproto graphite2 expat -y
+mamba install libgpg-error dbus -y
+mamba install libflac gettext -y
+mamba install xcb-util-wm xorg-libx11 xcb-util-image -y
+mamba install xkeyboard-config -y
+mamba install libxkbcommon fonts-conda-forge font-ttf-ubuntu gstreamer zlib -y
+mamba install xorg-xextproto libpng attr mpg123 -y
+mamba install pixman libvorbis glib-tools -y
+mamba install libsystemd0 xcb-util-keysyms xorg-libxrender libllvm15 -y
+mamba install font-ttf-dejavu-sans-mono pcre2 font-ttf-inconsolata font-ttf-source-code-pro -y
+mamba install lame nss xorg-xproto pthread-stubs xorg-libxdmcp -y
+mamba install libgcrypt xorg-libsm xorg-libxext fonts-conda-ecosystem xorg-kbproto mysql-libs -y
+mamba install fontconfig libjpeg-turbo xcb-util-renderutil -y
+mamba install glib -y
+mamba install freetype libcap libcups libopus -y
+mamba install gst-plugins-base mysql-common xcb-util -y
+mamba install cairo -y
+mamba install libsndfile harfbuzz xorg-libxau -y
+mamba install libglib libxcb -y
+mamba install qt-main -y
+mamba install pyqt -y
+mamba install matplotlib -y
+mamba install netcdf4 -y
+mamba install h5netcdf -y
+mamba install boto3 lxml -y
+mamba install scipy -y
+mamba install geos -y
+mamba install proj pyproj -y
+mamba install cartopy -y
+mamba install notebook -y
+mamba install progressbar -y
+mamba install gsw -y
+mamba install nco -y
+mamba install pympler -y
+
+# install remaining packages using pip
+# (mamba installs tend to get killed on t2.micro)
+pip install dask
+pip install "xarray[complete]"
+pip install jupyterlab
+pip install dask_labextension
+pip install s3fs
+pip install ecco_v4_py
+
+echo -e "${red_start}Completed Python package installations${nocolor_start}"
+
+
+echo -e "${red_start}Setting up NASA Earthdata authentication${nocolor_start}"
+# NASA Earthdata authentication
+# check if credentials are already archived in ~/.netrc, and if not then prompt the user for them
+earthdata_cred_stored=0
+if [ -f ~/.netrc ]; then
+    if grep -q "machine urs.earthdata.nasa.gov" ~/.netrc; then
+        earthdata_cred_stored=1
+        echo -e "${red_start}Earthdata credentials already archived ${nocolor_start}"
+    fi
+fi
+if [ $earthdata_cred_stored -eq 0 ]; then
+    if [ -f ~/.netrc ]; then sudo chmod 600 ~/.netrc; fi
+    read -p 'NASA Earthdata username: ' uservar
+    read -sp 'NASA Earthdata password: ' passvar
+    echo -e "machine urs.earthdata.nasa.gov\n    login ${uservar}\n    password ${passvar}\n" >> ~/.netrc
+    
+    echo -e "\n${red_start}NASA Earthdata authentication info archived in ~/.netrc${nocolor_start}"
+fi
+sudo chmod 400 ~/.netrc
+
+# create symlink to jupyter_lab_start.sh from the user's home directory
+ln -s ~/ECCO-v4-Python-Tutorial/Cloud_Setup/jupyter_lab_start.sh ~/jupyter_lab_start.sh
+
+echo "goodbye!"

Cloud_Setup/install_packages_bigmem.sh

@@ -0,0 +1,80 @@
+#!/bin/bash
+
+# Shell script for setting up conda, jupyter, essential Python packages on an AWS EC2 instance.
+
+# before installing, wise to update the local
+# machine with the latest packages. 
+# on redhat linux machines, 
+# $ sudo dnf update -y
+# $ sudo dnf install git -y
+
+# # Start body of script
+
+red_start='\033[0;31m'
+blue_start='\033[0;34m'
+nocolor_start='\033[0m'
+
+# # set paths to environment and package directories
+# create conda environment called 'jupyter'
+env_name='jupyter'
+
+conda create --name ${env_name} python=3.12 -y
+echo -e "${red_start}Created ${env_name} conda environment${nocolor_start}"
+
+# install python packages (using mamba) in jupyter environment
+conda activate ${env_name} 
+
+echo -e "${red_start}Installing Python packages in jupyter environment${nocolor_start}"
+
+# -n syntax installs the packages into the specified environment name
+mamba install -n ${env_name} requests tqdm numpy pandas xorg-libice libexpat libevent -y
+mamba install -n ${env_name} nspr alsa-lib libogg libpq xorg-renderproto xorg-xf86vidmodeproto graphite2 expat -y
+mamba install -n ${env_name} libgpg-error dbus libflac gettext xcb-util-wm xorg-libx11 xcb-util-image -y
+mamba install -n ${env_name} xkeyboard-config  libxkbcommon fonts-conda-forge font-ttf-ubuntu gstreamer zlib -y
+mamba install -n ${env_name} xorg-xextproto libpng attr mpg123 pixman libvorbis glib-tools -y
+mamba install -n ${env_name} libsystemd0 xcb-util-keysyms xorg-libxrender libllvm15 -y
+mamba install -n ${env_name} font-ttf-dejavu-sans-mono pcre2 font-ttf-inconsolata font-ttf-source-code-pro -y
+mamba install -n ${env_name} lame nss xorg-xproto pthread-stubs xorg-libxdmcp -y
+mamba install -n ${env_name} libgcrypt xorg-libsm xorg-libxext fonts-conda-ecosystem xorg-kbproto mysql-libs -y
+mamba install -n ${env_name} fontconfig libjpeg-turbo xcb-util-renderutil -y
+mamba install -n ${env_name} glib freetype libcap libcups libopus -y
+mamba install -n ${env_name} gst-plugins-base mysql-common xcb-util -y
+mamba install -n ${env_name} cairo libsndfile harfbuzz xorg-libxau -y
+mamba install -n ${env_name} libglib libxcb qt-main pyqt matplotlib netcdf4 -y
+mamba install -n ${env_name} h5netcdf boto3 lxml scipy goes proj pyproj cartopy notebook -y 
+mamba install -n ${env_name} progressbar gsw nco pympler -y
+mamba install -n ${env_name} xarray[complete] jupyterlab dask_labextension s3fs -y
+mamba install -n ${env_name} pyresample -y
+
+# install remaining packages using pip
+# (mamba installs tend to get killed on t2.micro)
+pip install ecco_v4_py --no-cache-dir
+
+echo -e "${red_start}Completed Python package installations${nocolor_start}"
+
+# Set up NASA Earthdata credential
+
+echo -e "${red_start}Setting up NASA Earthdata authentication${nocolor_start}"
+# NASA Earthdata authentication
+# check if credentials are already archived in ~/.netrc, and if not then prompt the user for them
+earthdata_cred_stored=0
+if [ -f ~/.netrc ]; then
+    if grep -q "machine urs.earthdata.nasa.gov" ~/.netrc; then
+        earthdata_cred_stored=1
+        echo -e "${red_start}Earthdata credentials already archived ${nocolor_start}"
+    fi
+fi
+if [ $earthdata_cred_stored -eq 0 ]; then
+    if [ -f ~/.netrc ]; then sudo chmod 600 ~/.netrc; fi
+    read -p 'NASA Earthdata username: ' uservar
+    read -sp 'NASA Earthdata password: ' passvar
+    echo -e "machine urs.earthdata.nasa.gov\n    login ${uservar}\n    password ${passvar}\n" >> ~/.netrc
+    
+    echo -e "\n${red_start}NASA Earthdata authentication info archived in ~/.netrc${nocolor_start}"
+fi
+sudo chmod 400 ~/.netrc
+
+# create symlink to jupyter_lab_start.sh from the user's home directory
+ln -s ~/ECCO-v4-Python-Tutorial/Cloud_Setup/jupyter_lab_start.sh ~/jupyter_lab_start.sh
+
+echo "goodbye!"

Cloud_Setup/install_wget_tmux.sh

@@ -0,0 +1,34 @@
+#!/bin/bash
+
+# script to install wget and tmux 
+# wget is needed to download Miniforge
+# tmux is useful (but not really necessary) for running jupyter 
+
+# # Start body of script
+
+red_start='\033[0;31m'
+blue_start='\033[0;34m'
+nocolor_start='\033[0m'
+
+if command -v dnf &> /dev/null
+then
+  # install wget and tmux
+  sudo dnf install wget tmux -y
+elif command -v apt &> /dev/null; then
+  sudo apt install wget tmux -y
+elif command -v yum &> /dev/null; then
+  sudo yum install wget tmux -y
+fi
+
+if command -v wget &> /dev/null; then
+  echo -e "${red_start}Installed wget${nocolor_start}"
+else
+  echo -e "${blue_start}Could not install wget! -- install it yourself ${nocolor_start}"
+fi
+
+if command -v tmux &> /dev/null; then
+  echo -e "${red_start}Installed tmux${nocolor_start}"
+else
+  echo -e "${blue_start}Could not install tmux! -- install it yourself ${nocolor_start}"
+fi
+