Skip to content

Commit 0bcf954

Browse files
authored
content: add dbr local (#60)
1 parent 635fc78 commit 0bcf954

6 files changed

Lines changed: 35 additions & 6 deletions

File tree

33.4 KB
Loading
63.8 KB
Loading
59.1 KB
Loading
105 KB
Loading
32.8 KB
Loading

app/content/posts/personal-website/local-dbr/local-dbr.mdx

Lines changed: 35 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,11 @@ thumb: ./assets/logo.svg
99
import { PostImage as Image } from "@/components/molecues/post-image";
1010

1111
import ImgLandscape from "./assets/landscape.svg";
12+
import ImgNewCluster from "./assets/new-cluster.png";
1213
import ImgSelectProfile from "./assets/select-profile.png";
14+
import ImgSelectEnv from "./assets/select-env.png";
15+
import ImgConfigureCluster from "./assets/configure-cluster.png";
16+
import ImageOutput from "./assets/output.png";
1317

1418
<Image src={ImgLandscape} />
1519

@@ -24,9 +28,20 @@ Before we get started, you'll need to have the following tools:
2428

2529
## Step 1: Create Cluster
2630

31+
The first step is to create a cluster in your Databricks workspace. That will be used to run your notebooks both in the workspace and locally.
32+
33+
1. Open your Databricks Portal and navigate to the `Compute` tab.
34+
2. Click on `Create Compute` to create a new cluster.
35+
3. Select runtime version >= 15 and configuration for your cluster. You can use the default settings for now.
36+
4. Click on `Create Compute` and wait for the cluster to be up.
37+
38+
After this step, you should have a new cluster running in your workspace.
39+
40+
<Image src={ImgNewCluster} />
41+
2742
## Step 2: Create Databricks Local Profile
2843

29-
The first step is to connect your Databricks workspace to your local machine and storing the connection details in a local profile. This will allow Databricks Extension for VSCode to connect to your workspace.
44+
In this step, we'll connect your Databricks workspace to your local machine and storing the connection details in a local profile. This will allow Databricks Extension for VSCode to connect to your workspace.
3045

3146
1. Install Databricks CLi by running one of the following commands:
3247

@@ -57,7 +72,7 @@ The first step is to connect your Databricks workspace to your local machine and
5772
databricks auth login --configure-cluster --host <workspace-url>
5873
```
5974

60-
Follows the prompts to log in and configure the cluster. After this step, you should be able to see the a DEFAULT PROFILE in `~/.databrickscfg` file.
75+
Follows the prompts to log in and configure the cluster. After this step, you should be able to see the a profile in `~/.databrickscfg` file.
6176

6277
## Step 3: Create Python Environment with Mini Conda
6378

@@ -124,6 +139,8 @@ Mini Conda is a lightweight version of Anaconda that allows you to create and ma
124139
pip install databricks-connect
125140
```
126141

142+
After this step, you should have a new python environment with `databricks-connect` installed.
143+
127144
## Step 4: Run notebooks locally
128145

129146
Now that we have our Databricks profile set up and a python environment ready, we can start working on our notebooks locally.
@@ -145,15 +162,21 @@ Now that we have our Databricks profile set up and a python environment ready, w
145162

146163
Now you should be able to see the Databricks icon in the sidebar of your VSCode.
147164

148-
3. Click on the Databricks icon in the sidebar and select `Configure Databricks`, then select `DEFAULT` profile we created in step 1.
165+
3. Click on the Databricks icon in the sidebar and select `Configure Databricks`, then select the profile we created in step 1.
149166

150167
<Image src={ImgSelectProfile} />
151168

152-
After this step, Databricks extension will create a folder ... as the destination for our folder.
169+
You can also enable sync, Databricks extension will push all the files in the local folder to the Databricks workspace.
153170

154171
**PLEASE NOTE THAT THIS IS ONE-WAY SYNC, ANY CHANGES MADE IN THE DATABRICKS WORKSPACE WILL NOT BE REFLECTED IN THE LOCAL FOLDER**.
155172

156-
4. Now you can create a new notebook for testing connection. Let's create a new notebook named `test.py` and add the following code:
173+
4. Also on the Databricks extension, click on `Configure cluster` and select the cluster we created in step 1. Don't forget to make sure the cluster is running.
174+
175+
<Image src={ImgConfigureCluster} />
176+
177+
5. Install Jupyter extension for VSCode to run the notebook. Visit [this link](https://marketplace.visualstudio.com/items?itemName=ms-toolsai.jupyter) and click on `Install`.
178+
179+
6. Now you can create a new notebook for testing connection. Let's create a new notebook named `test.py` and add the following code:
157180

158181
```python
159182
# COMMAND ----------
@@ -164,7 +187,13 @@ Now that we have our Databricks profile set up and a python environment ready, w
164187

165188
Don't forget include the `# COMMAND ----------` at the beginning of the cell and `from databricks.sdk.runtime import *` to import the necessary libraries.
166189

167-
Run the cell and you should see the output in the output cell.
190+
7. Open the command palette in VSCode by pressing `Ctrl + Shift + P` or `Cmd + Shift + P` and search for `Python: Select Interpreter`. Select the `dbr` environment we created in step 3.
191+
192+
<Image src={ImgSelectEnv} />
193+
194+
8. Run the notebook by pressing `Shift + Enter` or clicking on the `Run Cell` button and you should see the output in the output panel.
195+
196+
<Image src={ImageOutput} />
168197

169198
## Conclusion
170199

0 commit comments

Comments
 (0)