|
| 1 | +--- |
| 2 | +nanoId: np3i3m4dpqzxwi37u2lyo |
| 3 | +title: Databricks local development environment with VSCode and Mini Conda |
| 4 | +date: 2024-06-01 |
| 5 | +wip: false |
| 6 | +thumb: ./assets/logo.svg |
| 7 | +--- |
| 8 | + |
| 9 | +import { PostImage as Image } from "@/components/molecues/post-image"; |
| 10 | + |
| 11 | +import ImgLandscape from "./assets/landscape.svg"; |
| 12 | + |
| 13 | +<Image src={ImgLandscape} /> |
| 14 | + |
| 15 | +I recently started working with Databricks and wanted to set up a local development environment to make it easier to work with notebooks and version control. In this post, I'll take you through the steps to set up a local Databricks environment using VSCode and Mini Conda. |
| 16 | + |
| 17 | +## Prerequisites |
| 18 | + |
| 19 | +Before we get started, you'll need to have the following tools: |
| 20 | + |
| 21 | +- **Databricks workspace:** you can follow [this guide](https://docs.databricks.com/en/getting-started/index.html) to create get a trial workspace or use an existing one. |
| 22 | +- [Visual Studio Code](https://code.visualstudio.com/) installed on your machine. |
| 23 | + |
| 24 | +## Step 1: Conenct to Databricks workspace |
| 25 | + |
| 26 | +The first step is to connect your Databricks workspace to your local machine. You can do this by generating a token in your Databricks workspace and using it to connect to the workspace. |
| 27 | + |
| 28 | +## Step 3: Install Mini Conda |
| 29 | + |
| 30 | +Mini Conda is a lightweight version of Anaconda that allows you to create and manage python environments. It means you can have multiple python environments on your machine and switch between them easily. |
| 31 | + |
| 32 | +Anaconda offers graphical installers for Windows, macOS, and Linux but I would recommend using installation scripts. [You can find the OS specific installation scripts here](https://docs.anaconda.com/free/miniconda/#quick-command-line-install). At the time of writing, the installation script for **Linux** is as follows: |
| 33 | + |
| 34 | +```bash |
| 35 | +mkdir -p ~/miniconda3 |
| 36 | +wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh |
| 37 | +bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 |
| 38 | +rm -rf ~/miniconda3/miniconda.sh |
| 39 | +``` |
| 40 | + |
| 41 | +And don't forget to initialize mini conda for your shell, for example, for zsh: |
| 42 | + |
| 43 | +```bash |
| 44 | +~/miniconda3/bin/conda init zsh |
| 45 | +``` |
| 46 | + |
| 47 | +Now restart your shell and you should be able to use `conda` command. For example, I'm using zsh so I would run: |
| 48 | + |
| 49 | +```bash |
| 50 | +> zsh |
| 51 | + |
| 52 | +> conda --version |
| 53 | +conda 24.4.0 |
| 54 | +``` |
| 55 | + |
| 56 | +## Step 4: Prepare Python environment |
| 57 | + |
| 58 | +Next we'll create a new conda environment named `dbr` for our Databricks project. Run the following command to create a new environment: |
| 59 | + |
| 60 | +```bash |
| 61 | +> conda create -n dbr |
| 62 | +``` |
| 63 | + |
| 64 | +## Step 5: Run notebooks locally |
0 commit comments