|
1 | | -# Conda and installing Python and R environments |
| 1 | +# Conda |
2 | 2 |
|
3 | | -:::{admonition} Updates in progress |
4 | | -:class: warning |
5 | | - |
6 | | -The resources are actively being updated! Some parts are still out of date, and is the content from last year. In the meantime, please watch out for references to 2021 ("OHW21") or links that don't work. |
7 | | - |
8 | | -::: |
9 | | - |
10 | | -## Overview |
| 3 | +_or: How I Learned to Stop Worrying and Manage Python and R_ |
11 | 4 |
|
12 | | -### What is Conda? |
13 | | -[**Conda**](http://conda.pydata.org/docs/) is an **open source `package` and `environment` management system for any programming languages, but very popular among python community,** |
| 5 | +The JupyterHub is pre-configured with customized environments for both Python and R packages that are designed to be able to run all the tutorial notebooks, and support a broad range of oceanographic applications. |
14 | 6 |
|
15 | | -The Hub is pre-configured with a customized "environment" of Python and R packages designed to run all the tutorial notebooks, and supporting a broad range of oceanographic applications. This environment is created and managed using the open-source [**Conda** package and environment management system](https://docs.conda.io) for installing multiple versions of software packages together with their dependencies, and convenient switching between environments. Conda runs on Windows, macOS, and Linux: *"Conda quickly installs, runs, and updates packages and their dependencies. Conda easily creates, saves, loads, and switches between environments on your local computer. It was created for Python programs but it can package and distribute software for any language."* |
| 7 | +This environment is created and managed using the open-source [**Conda** package and environment management system](https://docs.conda.io) for installing multiple versions of software packages together with their dependencies, and convenient switching between environments. |
16 | 8 |
|
17 | | -Conda may be used on your computer as well as the Hub ... |
| 9 | +## What is Conda? |
| 10 | +[**Conda**](http://conda.pydata.org/docs/) is an **open source `package` and `environment` management system for any programming languages, but very popular among the Python community,** |
18 | 11 |
|
19 | | -https://github.com/oceanhackweek/ohw20-tutorials/blob/master/environment.yml |
| 12 | +Conda runs on Windows, macOS, and Linux: *"Conda quickly installs, runs, and updates packages and their dependencies. Conda easily creates, saves, loads, and switches between environments on your local computer. It was created for Python programs but it can package and distribute software for any language."* |
20 | 13 |
|
21 | 14 | For Python, the advantage of conda compared to `pip` is that it has a built in environment management system as well as the management of binaries, and non-Python dependencies. |
22 | 15 |
|
23 | | -You do not need administrative or root permissions to install conda if you select a user-writable installation location. |
24 | | - |
25 | | - |
26 | | -In the previous lesson we showed you a cloud-based environment for our work during the hackweek. What happens after the event when you want to go home and work with all the libraries we showed you? You will likely also want to have a functioning version of Python on your local laptop if that is not already in place. So this lesson takes you through our recommended procedure for doing that. We suggest you get this set up in advance so that we can help you troubleshoot when you arrive. |
27 | | - |
| 16 | +## Conda on the JupyterHub |
| 17 | + |
| 18 | +The JupyterHub has both a pre-configured base environment, and environments that you create and manage yourself. |
| 19 | + |
| 20 | +### JupyterHub base environment |
| 21 | + |
| 22 | +The Conda environment for the base JupyterHub environments are defined in [oceanhackweek/jupyter-image](https://github.com/oceanhackweek/jupyter-image/). These image contains hopefully everything you will need for the tutorials and for general exploration. |
| 23 | + |
| 24 | +The `environment.yml` files ([Python](https://github.com/oceanhackweek/jupyter-image/blob/main/py-base/environment.yml), [R](https://github.com/oceanhackweek/jupyter-image/blob/main/r/environment.yml)) captures the current state of the OceanHackWeek environment. You can explore these files to see what packages we have selected to come in the base environment. |
| 25 | + |
| 26 | +```yaml |
| 27 | +# environment.yml |
| 28 | +name: OHW |
| 29 | +channels: |
| 30 | + - conda-forge |
| 31 | +dependencies: |
| 32 | + - python=3.9 |
| 33 | + - pangeo-notebook=2021.07.24 |
| 34 | + - argopy |
| 35 | + - bokeh |
| 36 | + - bottleneck |
| 37 | + - cartopy |
| 38 | + - cdsapi |
| 39 | + - cf-units |
| 40 | + - cf_xarray |
| 41 | + - cmip6_preprocessing |
| 42 | + - cmocean |
| 43 | + - colorcet |
| 44 | + - compilers |
| 45 | + - compliance-checker |
| 46 | + - conda-lock |
| 47 | +# ... oh so many more packages that we are not going to include them all here |
| 48 | +``` |
28 | 49 |
|
29 | | -### Python Software |
| 50 | +It also contains a lot of supporting infrastructure for running each individual's JupyterLab server (for instance `compilers` and `conda-lock` in just that small subset), so we suggest building up an environment from scratch, rather than by trimming down the base environment. |
30 | 51 |
|
31 | | -Python software is distributed as a series of *libraries* that are called within your code to perform certain tasks. There are many different collections, or *distributions* of Python software. Generally you install a specific distribution of Python and then add additional libraries as you need them. There are also several different *versions* of Python. The two main versions right now are 2.7 and 3.7, although Python 2.7 will not be supported past 2020. Some libraries only work with specific versions of Python. |
| 52 | +The exact state of the Conda environments are captured in `conda-linux-64.lock` in the same directories that includes the exact versions of all the packages, not just the ones we selected. |
32 | 53 |
|
33 | | -So even though Python is one of the most adaptable, easy-to-use software systems, you can see there are still complexities to work out and potential challenges when delivering content to a large group. Therefore we have a number of different ways that we are trying to simplify this process to maximize your learning during the hackweek. |
| 54 | +There are also a handful of dependencies that are installed directly in the `Dockerfiles` that are also in the same directories. |
34 | 55 |
|
35 | | -We also provide instructions for using [Anaconda](https://www.continuum.io), which is our recommended Python distribution, for installing and working with Python on your local computer. We can assist in setting up "conda" environments that will simplify the gathering of Python libraries and version specific to the tutorial you are working on. |
| 56 | +The full environments are captured as [Docker images](https://github.com/orgs/oceanhackweek/packages?repo_name=jupyter-image) that can be pulled and run locally. |
36 | 57 |
|
| 58 | +### Temporary packages |
37 | 59 |
|
38 | | -## Installing Conda Miniconda |
| 60 | +You can temporarily add packages to your hub, via Jupyter cell magic, `%pip install <list-of-packages>` or `%conda install <list-of-packages>`. In R you can use `install.packages("package-name")` as usual. |
39 | 61 |
|
40 | | -:::{admonition} For local development |
41 | | -:class: warning |
| 62 | +:::{admonition} pip install trouble |
| 63 | +:class: danger |
42 | 64 |
|
43 | | -Conda is already installed on our JupyterHub, so these instructions are for if you wish to get started with developing locally. |
44 | | - |
45 | | -We may not have the ability to support everyone's individual system, so we have the JupyterLab setup so that everyone can work on the same pre-configured platform. |
| 65 | +For those who know their way around Jupyter, you may be tempted to `!pip install <list-of-packages>`. This can leave your environment in an inconsistent state, which may prevent your server from starting (and will require some heavy duty assistance from `@help-infrastructure` to debug). More information is [available here.](https://pilot.2i2c.org/en/latest/admin/howto/environment.html#temporarily-install-packages-for-a-session) |
46 | 66 |
|
47 | 67 | ::: |
48 | 68 |
|
49 | | -If you don't have conda (either with *Miniconda* or the full *Anaconda Distribution*) already installed **we recommend [installing Miniconda for latest Python 3](https://docs.conda.io/en/latest/miniconda.html).** |
| 69 | +### Create your own environment on JupyterHub |
50 | 70 |
|
51 | | -https://conda.io/projects/conda/en/latest/user-guide/install/index.html |
| 71 | +To create your own Conda environment on JupyterHub, you can launch the terminal and run `conda create` commands as expected. Be sure to specify `-n <environment-name>`. For a Python environment: |
52 | 72 |
|
53 | | -### Windows |
| 73 | +`conda create -n cool-project -c conda-forge python=3.9 xarray ipykernel` |
54 | 74 |
|
55 | | -Download the proper installer for your Windows platform (64 bits). When installing, you will be asked if you wish to make the Anaconda Python your default Python for Windows. If you do not have any other installation that is a good option. If you want to keep multiple versions of Python on your machine (e.g. ESRI-supplied python, or 64 bit versions of Anaconda), then don't select the option to modify your path or modify your Windows registry settings. |
| 75 | +:::{admonition} Kernel needed |
| 76 | +:class: warning |
56 | 77 |
|
57 | | -### Linux and OSX |
| 78 | +In order to get easy notebook or terminal access in JupyterLab, a Jupyter kernel needs to be included in the environment, such as `ipykernel` for Python or `irkernel` for R. |
58 | 79 |
|
59 | | -You may follow manual steps from [here](https://docs.conda.io/en/latest/miniconda.html) similar to the instructions on Windows (see above). Alternatively, you can execute these commands on a terminal shell (in this case, the bash shell): |
| 80 | +::: |
60 | 81 |
|
61 | | -```bash |
62 | | -# For MacOSX |
63 | | -url=https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh |
64 | | -# For Linux |
65 | | -url=https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh |
66 | | -wget $url -O miniconda.sh |
67 | | -bash miniconda.sh -b -p $HOME/miniconda |
68 | | -export PATH="$HOME/miniconda/bin:$PATH" |
69 | | -conda update conda --yes |
70 | | -``` |
| 82 | +Once you've created an environment, you can run `conda activate cool-project` as usual for access to the environment in the terminal. |
71 | 83 |
|
72 | | -## Installing Python |
| 84 | +:::{admonition} Wait for it... |
73 | 85 |
|
74 | | -We will be using Python 3.8 or 3.9 during the week (either will work). Since Anaconda (on Linux) expects you to work in the "bash" shell, if this is not already your default shell, you need to set it to be so (use the "chsh -s /bin/bash" command to change your default shell to bash) or just run an instance of bash from the command line before issuing "Conda" commands (/bin/bash or where it is located on your system). |
| 86 | +It may take a minute or two for JupyterLab to show your new Conda environment. |
| 87 | +The [package](https://github.com/Anaconda-Platform/nb_conda_kernels) that detects additional environments doesn't run constantly, so give it a second before worrying that you created an environment wrong. |
75 | 88 |
|
76 | | -If you are already familiar with Python 2.7, you can take a look at the syntax differences [here](http://sebastianraschka.com/Articles/2014_python_2_3_key_diff.html), but the main point to remember is to put the print statements in parentheses: |
| 89 | +::: |
77 | 90 |
|
78 | | -```python |
79 | | -print('Hello World!') |
80 | | -``` |
| 91 | +## Conda on your own computer |
81 | 92 |
|
82 | | -``` bash |
83 | | -$ conda create -n py39 python=3.9 |
84 | | -``` |
| 93 | +Conda may be used on your computer as well as the Hub. If you wish to install the same environment as the hub is running, after you install Conda, you can download the [`environment.yml`](https://github.com/oceanhackweek/ohw20-tutorials/blob/master/environment.yml) that we use, then `conda create -n <ENV NAME> --file environment.yml` |
85 | 94 |
|
86 | | -To use Python 3.9: |
| 95 | +### Installing Conda |
87 | 96 |
|
88 | | -``` bash |
89 | | -$ conda activate py39 |
90 | | -``` |
| 97 | +There are a few different ways to install conda: |
91 | 98 |
|
92 | | -To check if you have the correct version: |
| 99 | +- The [Anaconda Individual Edition](https://www.anaconda.com/products/individual) which comes with a large pre-packaged environment, and a snazzy management interface to help explore what packages are available and what environments you have installed. |
| 100 | +- [Miniconda](https://docs.conda.io/en/latest/miniconda.html) is a stripped down version with just the installer, which is really for kick starting other environments. We're using a Miniconda Docker image. |
| 101 | +- There is also [Mamba](https://mamba.readthedocs.io/en/latest/index.html) which is a newer take on Conda that tends to be faster, but isn't currently compatible with our trick to allow you to set up your own Conda environments ([nb_conda_kernels](https://github.com/Anaconda-Platform/nb_conda_kernels)). |
93 | 102 |
|
94 | | -``` bash |
95 | | -$ python --version |
96 | | -``` |
| 103 | +We recommend the use of Miniconda. |
0 commit comments