Update Tutorial to new MLBench version

Ralf Grubenmann · Ralf Grubenmann · commit 00777d5f2dd0 · 2019-02-07T14:02:59.000+01:00
diff --git a/_posts/2018-09-10-tutorial.md b/_posts/2018-09-10-tutorial.md
@@ -17,166 +17,55 @@ This tutorial guides you through setting up MLBench in a Google Cloud [Kubernete
 This tutorial assumes you have a Google Cloud account with permissions to create a new cluster.
 You also need to have [Python](https://www.python.org/), [Git](https://git-scm.com/), and [Docker](https://www.docker.com) installed locally and the Docker Daemon should be running.
 
-Now you have to checkout the mlbench github repository and have a terminal open in the checked-out mlbench directory.
+Checkout the [mlbench-helm](https://github.com/mlbench/mlbench-helm)) github repository and have a terminal open in the checked-out mlbench directory.
 
 ```shell
-$ git clone git@github.com:mlbench/mlbench.git
+$ git clone git@github.com:mlbench/mlbench-helm.git
 ```
 
 Enter the newly created directory
 
 ```shell
-$ cd mlbench
+$ cd mlbench-helm
 ```
 
-### Setting up gcloud client
-
-Follow the steps detailed [here](https://cloud.google.com/sdk/docs/quickstarts) to install the Google Cloud SDK.
-
-Now install the [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/) tool with the Google configuration. kubectl is a command line interface for communicating with a Kubernetes API server.
-
-```shell
-$ gcloud components install kubectl
-```
-
-This will configure the kubernetes kubectl with the correct credentials for your account.
-
-We can now create a Kubernetes cluster called ``mlbench`` by running
-
-```shell
-$ gcloud container clusters create mlbench --machine-type='n1-standard-2'
-```
-By default, this will create a new cluster with 3 nodes, all of which are ``n1-standard-2`` instances.
-Once the cluster is created, we need to set the correct credentials for kubectl
-
-```shell
-$ gcloud container clusters get-credentials mlbench
-```
-
-This sets the default context of kubectl to our newly created cluster.
-
-### Installing Helm
-
-[Helm](https://github.com/helm/helm/) is a package manager for Kubernetes applications. It helps install pre-defined distributed applications to clusters.
-
-To install helm, run
-
-```shell
-$ curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash
-```
-
-For helm to work properly, it needs a service account in the cluster with ``cluster-admin`` rights. We can set up an account with the correct privileges by running
-
-```shell
-$ kubectl --namespace kube-system create sa tiller
-$ kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
-```
-
-This creates a new service account with the correct privileges for the helm server component ``tiller``, which takes care of managing the deployment of pods to our cluster.
-
-We can now initialize helm with our newly created service account
-
-```shell
-$ helm init --service-account tiller
-```
-
-After this, helm is set up and ready to deploy applications to our newly created cluster.
-
-### Building the Master and Worker images
-
-*Note: You can skip this part if you want to use the precompiled docker images*
-
-To use custom images, we will have to host them in a docker registry. The [Google Cloud Container Registry](https://cloud.google.com/container-registry/) is an obvious choice for Google Cloud.
-
-First we need to enable access to it on our commandline
-
-```shell
-$ gcloud auth configure-docker
-```
-
-We provide easy to use commands to build and deploy the images using ```make```. Make sure to correctly set the name of your Google Cloud project (``<gcloud project name>``) in the following commands
-
-```shell
-$ make publish-docker component=master docker_registry=gcr.io/<gcloud project name>
-$ make publish-docker component=worker docker_registry=gcr.io/<gcloud project name>
-```
-
-This will build the ``master`` and ``worker`` docker images and push them to the Container Registry.
-
 ### Installing MLBench
 
-Now copy the file `charts/mlbench/values.yaml` to the current directory, calling it `myvalues.yaml`.
+Copy the file `values.yaml` to the current directory, calling it `myvalues.yaml`.
 
 ```shell
-$ cp charts/mlbench/values.yaml myvalues.yaml
+$ cp values.yaml myvalues.yaml
 ```
 
 This file contains default values for most settings in mlbench. There are however some you need to set yourself to reasonable values for your cluster, namely:
 
 ```yaml
 limits:
   cpu: 1000m
-  gpu: 0
-  maximumWorkers: 3
+  workers: 3
   bandwidth: 1000
-```
-
-This limits the maximum usable resources (And the maximum you are able to chose in the UI) to 1 CPU core , 0 GPUs, 1000 mbit/s network speed per node and 3 nodes total.
-
-*Note: Our ``n1-standard-2`` instances have 2 CPU cores. But due to Google Cloud Kubernetes running its own monitoring and management pods, which also use some CPU, it is advisable to set MLBench to use one core less than available*
-
-If you followed the previous section and built the docker images yourself, your ``myvalues.yaml`` file should look as follows (again, replace ``<gcloud project name>`` with your Google Cloud project name)
-
-```yaml
-master:
-  image:
-    repository: gcr.io/<gcloud project name>/mlbench_master
-    tag: latest
-    pullPolicy: Always
-
-
-worker:
-  image:
-    repository: gcr.io/<gcloud project name>/mlbench_worker
-    tag: latest
-    pullPolicy: Always
-
-limits:
-  cpu: 1000m
   gpu: 0
-  maximumWorkers: 3
-  bandwidth: 1000
 ```
 
-Now it is time to install MLBench
-
-```shell
-$ helm upgrade --wait --recreate-pods -f myvalues.yaml --timeout 900 --install release1 charts/mlbench
-```
+This limits the maximum usable resources (And the maximum you are able to chose in the UI) to 1 CPU (1000m = 1000 milli-CPUs) core , 0 GPUs, 1000 mbit/s network speed per node and 3 nodes total.
 
-This creates Kubernetes templates based on the values set in ``myvalues.yaml`` and installs them to our Kubernetes cluster, calling the release ``release1``.
+*Note: ``n1-standard-2`` instances have 2 CPU cores. But due to Google Cloud Kubernetes running its own monitoring and management pods, which also use some CPU, it is advisable to set MLBench to use one core less than available*
 
-*Note: Release names allow you to install multiple instances of the same helm chart side by side, but are not relevant for this tutorial*
+With those values set, MLBench can be installed with the `google_cloud_setup.sh` script (Run `google_cloud_setup.sh help` to see all available options).
 
-Since the deployment is not open to the internet by default, the default instructions printed by the previous command **do not apply**.
-To gain access to MLBench, we need to add a firewall rule to Google Cloud
+First, create a GKE cluster:
 
 ```shell
-$ export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services ${RELEASE_NAME}-mlbench-master)
-$ export NODE_IP=$(gcloud compute instances list|grep $(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}") |awk '{print $5}')
-$ gcloud compute firewall-rules create --quiet mlbench --allow tcp:$NODE_PORT,tcp:$NODE_PORT
+$ ./google_cloud_setup.sh create-cluster NUM_NODES=4
 ```
 
-This gets the public ip of the node the ``master`` image is deployed on, plus the randomly selected port it is running on, and adds a firewall rule allowing access to that port.
-
-To get the URL the dashboard is accessible under, we can now just run
+and then install the helm chart:
 
 ```shell
-$ echo http://$NODE_IP:$NODE_PORT
-http://172.16.0.1:32145
+$ ./google_cloud_setup.sh install-chart
 ```
 
-and it should print the URL (In this example it printed ``http://172.16.0.1:32145``)
+That's it, this should setup MLBench in your Google Kubernetes cluster. The Dashboard URL can be found in at the end of the output of the last command (e.g. `http://172.16.0.1:32145`).
 
 Simply open the URL in your browser and you should be ready to go.
 
@@ -222,15 +111,21 @@ You can then see the details of the experiment by clicking on its entry in the l
 
 That's it! You successfully ran an distributed machine learning algorithm in the cloud. You can also easily develop custom worker images for your own models and compare them to existing benchmarking code without a lot of overhead.
 
-### Appendix 1: Run mlbench on GKE
 
-The previous commands can be summarized as follows
+### Cleanup
+To delete MLBench, run :
+
+```shell
+$ ./google_cloud_setup.sh uninstall-chart
+```
 
-{% gist 23361aea5fe252570496acc7da4fb599 %}
+To delete the whole Cluster (and cleanup firewall rules), run:
 
-Customize environment variables like `NUM_NODES` and run the scripts with `create`, `install` and `dashboard` sequantially. The last command will give you the external ip of the dashboard. When the job is done, run this script with `cleanup` to delete everything.
+```shell
+$ ./google_cloud_setup.sh delete-cluster
+```
 
-### Appendix 2: Use NFS for Data storage
+### Appendix 1: Use NFS for Data storage
 To avoid downloading datasets everytime we reinstall mlbench, we can use a persistent disk to save the data. To do so, one can create a GCE disk like
 ```bash
 gcloud compute disks create --size=10G --zone=europe-west1-b my-pd-name
