Skip to content

Commit ddea0c4

Browse files
fedorovfedorov
committed
initial commit of DICOM store creation
1 parent a5607ce commit ddea0c4

1 file changed

Lines changed: 344 additions & 0 deletions

File tree

‎notebooks/viewers_deployment/Creating_Google_Healthcare_DICOM_store.ipynb‎

Lines changed: 344 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,344 @@
1+
{
2+
"nbformat": 4,
3+
"nbformat_minor": 0,
4+
"metadata": {
5+
"colab": {
6+
"provenance": [],
7+
"private_outputs": true,
8+
"toc_visible": true,
9+
"include_colab_link": true
10+
},
11+
"kernelspec": {
12+
"name": "python3",
13+
"display_name": "Python 3"
14+
}
15+
},
16+
"cells": [
17+
{
18+
"cell_type": "markdown",
19+
"metadata": {
20+
"id": "view-in-github",
21+
"colab_type": "text"
22+
},
23+
"source": [
24+
"<a href=\"https://colab.research.google.com/github/ImagingDataCommons/IDC-Tutorials/blob/master/notebooks/viewers_deployment/Creating_Google_Healthcare_DICOM_store.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
25+
]
26+
},
27+
{
28+
"cell_type": "markdown",
29+
"source": [
30+
"## Prerequisites\n",
31+
"\n",
32+
"This notebook complements the tutorial available in this document: https://tinyurl.com/idc-ohif-gcp\n",
33+
"\n",
34+
"You must complete the prerequisites discussed in that document before proceeding with this notebook!\n",
35+
"\n",
36+
"Please use the document above for providing feedback or asking questions!"
37+
],
38+
"metadata": {
39+
"id": "ScmfC87l5FAZ"
40+
}
41+
},
42+
{
43+
"cell_type": "markdown",
44+
"metadata": {
45+
"id": "2tuVb1sCCjUV"
46+
},
47+
"source": [
48+
"## Preparation\n",
49+
"\n",
50+
"You will need to authenticate with Google to be able to complete this notebook."
51+
]
52+
},
53+
{
54+
"cell_type": "code",
55+
"metadata": {
56+
"id": "RVxG6QvteybL"
57+
},
58+
"source": [
59+
"from google.colab import auth\n",
60+
"auth.authenticate_user()"
61+
],
62+
"execution_count": null,
63+
"outputs": []
64+
},
65+
{
66+
"cell_type": "markdown",
67+
"source": [
68+
"Substitute `REPLACE_WITH_YOUR_PROJECT_ID` with the ID of the project you can access, which has billing configured, in the following cell and anywhere else you may see it."
69+
],
70+
"metadata": {
71+
"id": "DUFqJyhx45MC"
72+
}
73+
},
74+
{
75+
"cell_type": "code",
76+
"source": [
77+
"import os\n",
78+
"os.environ[\"MY_PROJECT_ID\"] = \"REPLACE_WITH_YOUR_PROJECT_ID\""
79+
],
80+
"metadata": {
81+
"id": "IjqLpKMWXOKi"
82+
},
83+
"execution_count": null,
84+
"outputs": []
85+
},
86+
{
87+
"cell_type": "code",
88+
"source": [
89+
"!gcloud config set project ${MY_PROJECT_ID}"
90+
],
91+
"metadata": {
92+
"id": "PtwyS8d2c9ho"
93+
},
94+
"execution_count": null,
95+
"outputs": []
96+
},
97+
{
98+
"cell_type": "markdown",
99+
"metadata": {
100+
"id": "6is-ni21Ac1_"
101+
},
102+
"source": [
103+
"## Define the list of instances\n",
104+
"\n",
105+
"We will use the query below to get the URLs for the files corresponding to the DICOM instances included in the sample DICOM study.\n",
106+
"\n",
107+
"Same as in the previous step, make sure to replace `REPLACE_WITH_YOUR_PROJECT_ID` in the cell below with the ID of your project, which should have billing enabled."
108+
]
109+
},
110+
{
111+
"cell_type": "code",
112+
"metadata": {
113+
"id": "9rsNuDaMB6wO"
114+
},
115+
"source": [
116+
"%%bigquery --project=REPLACE_WITH_YOUR_PROJECT_ID cohort_df\n",
117+
"\n",
118+
"SELECT gcs_url\n",
119+
"FROM `bigquery-public-data.idc_current.dicom_all`\n",
120+
"WHERE StudyInstanceUID = \"1.3.6.1.4.1.14519.5.2.1.6279.6001.224985459390356936417021464571\""
121+
],
122+
"execution_count": null,
123+
"outputs": []
124+
},
125+
{
126+
"cell_type": "markdown",
127+
"metadata": {
128+
"id": "vDSBOmiUDpxV"
129+
},
130+
"source": [
131+
"## Retrieve DICOM instances\n",
132+
"\n",
133+
"First, we will save the Google Cloud Storage (GCS) URLs into a separate file."
134+
]
135+
},
136+
{
137+
"cell_type": "code",
138+
"metadata": {
139+
"id": "bife6XMuhWTD"
140+
},
141+
"source": [
142+
"cohort_df = cohort_df.join(cohort_df[\"gcs_url\"].str.split('#', 1, expand=True).rename(columns={0:'gcs_url_no_revision', 1:'gcs_revision'}))\n",
143+
"cohort_df[\"gcs_url_no_revision\"].to_csv(\"gcs_paths.txt\", header=False, index=False)"
144+
],
145+
"execution_count": null,
146+
"outputs": []
147+
},
148+
{
149+
"cell_type": "code",
150+
"metadata": {
151+
"id": "5kXtIUh4hzO0"
152+
},
153+
"source": [
154+
"!head /content/gcs_paths.txt"
155+
],
156+
"execution_count": null,
157+
"outputs": []
158+
},
159+
{
160+
"cell_type": "markdown",
161+
"metadata": {
162+
"id": "oqP67MonHBam"
163+
},
164+
"source": [
165+
"Next, we will download the files to the VM filesystem using the standard `gsutil` command, which is preinstalled on Colab instances.\n",
166+
"\n",
167+
"IDC-hosted data is available from free Google Storage buckets maintained under [Google Public Dataset Program](https://console.cloud.google.com/marketplace/product/gcp-public-data-idc/nci-idc-data), which sponsors free egress of the data either within or out of the Google Cloud."
168+
]
169+
},
170+
{
171+
"cell_type": "code",
172+
"metadata": {
173+
"id": "BCjgz_DTnlgX"
174+
},
175+
"source": [
176+
"# https://cloud.google.com/storage/docs/gsutil/commands/cp\n",
177+
"%%capture\n",
178+
"!mkdir downloaded_cohort\n",
179+
"!cat gcs_paths.txt | gsutil -m cp -I ./downloaded_cohort"
180+
],
181+
"execution_count": null,
182+
"outputs": []
183+
},
184+
{
185+
"cell_type": "markdown",
186+
"metadata": {
187+
"id": "L_lE0c1FIvha"
188+
},
189+
"source": [
190+
"## Create and populate GCP Healthcare DICOM store\n",
191+
"\n",
192+
"Next we will create a temporary storage bucket that will contain the files that will be imported into a DICOM store. Note that we could import the files directly, one by one, from their original locations, but that operation is slower."
193+
]
194+
},
195+
{
196+
"cell_type": "code",
197+
"source": [
198+
"os.environ[\"MY_TEMP_BUCKET\"] = \"gs://\"+os.environ[\"MY_PROJECT_ID\"]+\"_af_ohiftutorial_temp\"\n",
199+
"print(\"DICOM data bucket location: \"+os.environ[\"MY_TEMP_BUCKET\"])"
200+
],
201+
"metadata": {
202+
"id": "IXo205RjZv5X"
203+
},
204+
"execution_count": null,
205+
"outputs": []
206+
},
207+
{
208+
"cell_type": "markdown",
209+
"source": [
210+
"**WARNING**: storage bucket names must be globally unique! If you fail to create a bucket due to a conflict, choose a different name, or reuse an existing bucket!"
211+
],
212+
"metadata": {
213+
"id": "rqMQKvkSdO9G"
214+
}
215+
},
216+
{
217+
"cell_type": "code",
218+
"source": [
219+
"!gsutil mb -p $MY_PROJECT_ID ${MY_TEMP_BUCKET}"
220+
],
221+
"metadata": {
222+
"id": "O744yoSdW4AH"
223+
},
224+
"execution_count": null,
225+
"outputs": []
226+
},
227+
{
228+
"cell_type": "code",
229+
"metadata": {
230+
"id": "sADXT6tukTu2"
231+
},
232+
"source": [
233+
"%%capture\n",
234+
"!gsutil -m cp -r ./downloaded_cohort/* ${MY_TEMP_BUCKET}"
235+
],
236+
"execution_count": null,
237+
"outputs": []
238+
},
239+
{
240+
"cell_type": "markdown",
241+
"source": [
242+
"In the following cells we are first creating a Healthcare dataset, and then a DICOM store within that dataset.\n",
243+
"\n",
244+
"**WARNING**:\n",
245+
"* please make sure Google Cloud Healthcare Service Agent has the Storage Viewer role to access the storage bucket! If you are not sure how to assign that role, please see the main tutorial document [here](https://docs.google.com/document/d/1v4Syu_yOV6yH--QBLGzsL9fJ7-XyD1CnQu4iTIoPVD8/edit#heading=h.jg76iucxtly9).\n",
246+
"* if the dataset or DICOM store with the given names already exist, you can skip the following steps, or choose different names for those resources."
247+
],
248+
"metadata": {
249+
"id": "eC_kXT9jZHbW"
250+
}
251+
},
252+
{
253+
"cell_type": "code",
254+
"source": [
255+
"# These uniquely identify your DICOM store:\n",
256+
"# you will need those if you want to open content of this store in OHIF Viewer\n",
257+
"os.environ[\"DATASET_ID\"] = \"af_ohiftutorial_temp_dataset\"\n",
258+
"os.environ[\"STORE_ID\"] = \"af_ohiftutorial_temp_store\"\n",
259+
"os.environ[\"LOCATION\"] = \"us-central1\""
260+
],
261+
"metadata": {
262+
"id": "USSz27O4aV-3"
263+
},
264+
"execution_count": null,
265+
"outputs": []
266+
},
267+
{
268+
"cell_type": "code",
269+
"source": [
270+
"!gcloud healthcare datasets create ${DATASET_ID} --project=${MY_PROJECT_ID} --location=${LOCATION}"
271+
],
272+
"metadata": {
273+
"id": "-utc1cmGclST"
274+
},
275+
"execution_count": null,
276+
"outputs": []
277+
},
278+
{
279+
"cell_type": "code",
280+
"source": [
281+
"!gcloud healthcare dicom-stores create ${STORE_ID} --project=${MY_PROJECT_ID} --dataset=${DATASET_ID} --location=${LOCATION}"
282+
],
283+
"metadata": {
284+
"id": "crCPOQK-aFlo"
285+
},
286+
"execution_count": null,
287+
"outputs": []
288+
},
289+
{
290+
"cell_type": "markdown",
291+
"source": [
292+
"Finally, now that we have a GCS storage buckets with the files we want to use to populate the DICOM store, we can import the content of the bucket into the DICOM store created in the previous step."
293+
],
294+
"metadata": {
295+
"id": "rfrHJUch8Yeg"
296+
}
297+
},
298+
{
299+
"cell_type": "code",
300+
"source": [
301+
"!gcloud healthcare dicom-stores import gcs ${STORE_ID} \\\n",
302+
" --dataset=${DATASET_ID} \\\n",
303+
" --location=${LOCATION} \\\n",
304+
" --gcs-uri=${MY_TEMP_BUCKET}/**.dcm \\\n",
305+
" --project=${MY_PROJECT_ID}"
306+
],
307+
"metadata": {
308+
"id": "8gE7zbsldQHp"
309+
},
310+
"execution_count": null,
311+
"outputs": []
312+
},
313+
{
314+
"cell_type": "markdown",
315+
"source": [
316+
"## Viewing DICOM store content in OHIF Viewer\n",
317+
"\n",
318+
"If you completed deployment of the OHIF Viewer, as described in [the tutorial document](https://docs.google.com/document/d/1v4Syu_yOV6yH--QBLGzsL9fJ7-XyD1CnQu4iTIoPVD8/edit#heading=h.jg76iucxtly9), you should now be able to open the study we retrieved in the first steps of this notebook in the viewer.\n",
319+
"\n",
320+
"To do this, update the string below to form the URL:\n",
321+
"\n",
322+
"* MY_OHIF_VIEWER: Firebase hosting URL\n",
323+
"* MY_PROJECT_ID: project under which you created the Healthcare dataset\n",
324+
"* DATASET_ID: dataset ID you specified earlier while creating the dataset\n",
325+
"* STORE_ID: DICOM store ID you specified earlier\n",
326+
"* 1.3.6.1.4.1.14519.5.2.1.6279.6001.224985459390356936417021464571 is the DICOM StudyInstanceUID that we used in the beginning of this notebook to retrieve the study from IDC - you can replace this with the StudyInstanceUID corresponding to any study you have in the DICOM store.\n",
327+
"\n",
328+
"**MY_OHIF_VIEWER_URL/projects/MY_PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/dicomStores/STORE_ID/study/1.3.6.1.4.1.14519.5.2.1.6279.6001.224985459390356936417021464571**"
329+
],
330+
"metadata": {
331+
"id": "Ca1EksVyfe22"
332+
}
333+
},
334+
{
335+
"cell_type": "markdown",
336+
"source": [
337+
"Now is the good time to return back to the [tutorial document](https://docs.google.com/document/d/1v4Syu_yOV6yH--QBLGzsL9fJ7-XyD1CnQu4iTIoPVD8/edit#heading=h.jg76iucxtly9)!"
338+
],
339+
"metadata": {
340+
"id": "WtvtwQM1hPwd"
341+
}
342+
}
343+
]
344+
}

0 commit comments

Comments
 (0)