updating to address suggestions from @bcli4d

Author: fedorov

notebooks/getting_started/part1_prerequisites.ipynb

@@ -38,6 +38,15 @@
         "[NCI Imaging Data Commons (IDC)](https://datacommons.cancer.gov/repository/imaging-data-commons) is a cloud-based repository of publicly available cancer imaging data co-located with the analysis and exploration tools and resources. IDC is a node within the broader NCI Cancer Research Data Commons (CRDC) infrastructure that provides secure access to a large, comprehensive, and expanding collection of cancer research data."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## What is \"IDC collection\"?\n",
+        "\n",
+        "IDC contains images and image-derived data (i.e., annotations and analysis results) from a variety of repositories. Those images are broadly organized by programs, which correspond to the various data collection initiatives. Programs consist of collections, which group data collected by a specific entity for a specific application. Collections include both original data that was collected by the contributing entity, and the image-derived data that might have been generated by other contributors, extending the original collection."
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {},
@@ -118,6 +127,33 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
+        "## Locate and add `bigquery-public-data` project\n",
+        "\n",
+        "`bigquery-public-data` is a public project that contains BigQuery tables with IDC metadata (we will work with those in the part 2 of this series). To tavigate those metadata tables we need to manually add this project.\n",
+        "\n",
+        "1. Navigate to the BigQuery console: https://console.cloud.google.com/bigquery, and click the `+ ADD DATA` button.\n",
+        "\n",
+        "![add data](https://www.dropbox.com/s/cg99cyn1uzigw7s/add_data.png?raw=1)\n",
+        "\n",
+        "2. Choose \"Star a project\" option from the list.\n",
+        "\n",
+        "![star a project](https://www.dropbox.com/s/6688galhthr5vsn/star_a_project.png?raw=1)\n",
+        "\n",
+        "3. Type `bigqeury-public-data` as the project name and click `STAR` button.\n",
+        "\n",
+        "![star](https://www.dropbox.com/s/nzh7aybkre138g1/star.png?raw=1)\n",
+        "\n",
+        "In a few moments, `bigquery-public-data` project should appear in the list on the left hand side of the BigQuery console.\n",
+        "\n",
+        "![starred](https://www.dropbox.com/s/s2f6vpolbimnyb8/bqpd_added.png?raw=1)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Check the setup\n",
+        "\n",
         "Finally, let's run a query to confirm that the setup is working for your account."
       ]
     },

notebooks/getting_started/part2_searching_basics.ipynb

@@ -148,6 +148,29 @@
         "In this query we work with the [`dicom_all` table](https://console.cloud.google.com/bigquery?p=bigquery-public-data&d=idc_current&t=dicom_all&page=table), which contains the DICOM metadata extracted from IDC images along with collection-level metadata that does not originate from DICOM."
       ]
     },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Organization of IDC metadata in BigQuery tables\n",
+        "\n",
+        "Let's take a moment to look into the table used in the `FROM` clause of our query: [`bigquery-public-data.idc_current.dicom_all`](https://console.cloud.google.com/bigquery?p=bigquery-public-data&d=idc_current&t=dicom_all&page=table).\n",
+        "\n",
+        "This name is like an address that allows to locate the specific table in BigQuery. This \"address\" consists of three components: <project_id>.<dataset_id>.<table_id>\n",
+        "\n",
+        "1. `bigquery-public-data` is a public GCP _project_ that is maintained by Google Public Datasets Program. IDC-curated BigQuery tables with the metadata about IDC images is included in this project.\n",
+        "2. `idc_current`is a _dataset_ within the `bigquery-public-data` project. Think of BigQuery datasets as containers that are used to organize and control access to the tables within the project.\n",
+        "3. `dicom_all` is one of the tables within the `idc_current` dataset. As you spend more time learning about IDC, you will hopefully leverage other tables available in that dataset.\n",
+        "\n",
+        "If you now look back at the [BigQuery console](https://console.cloud.google.com/bigquery) and expand the list of datasets under the `bigquery-public-data` project, you will see that in addition to the `idc_current` dataset there are also datasets `idc_v12`, `idc_v11`, etc all the way to `idc_v1`. Those datasets correspond to the IDC data release versions, with `idc_current` being an alias for the latest (at the moment, v12) version of IDC data. \n",
+        "\n",
+        "We will not spend time discussing how IDC versioning works, but it is important to know that \n",
+        "\n",
+        "1. IDC data is versioned;\n",
+        "2. queries against the `idc_current` dataset are equivalent to the queries against the latest version (currently, `idc_v12`) of IDC data;\n",
+        "3. if you want the results of the queries to be persistent, write those against `idc_v*` datasets instead of `idc_current`."
+      ]
+    },
     {
       "cell_type": "markdown",
       "metadata": {
@@ -345,6 +368,9 @@
         "  bigquery-public-data.idc_current.dicom_all\n",
         "WHERE\n",
         "  # write the selection criteria under this line!\n",
+        "  # Use AND operator to combine the filter values for the\n",
+        "  # Modality and tcia_tumorLocation to select collections that\n",
+        "  # include MR images for Lung cancer locations\n",
         "\"\"\"\n",
         "\n",
         "selection_result = bq_client.query(selection_query)\n",
@@ -615,7 +641,9 @@
         "* learned about BigQuery as the tool for searching IDC metadata\n",
         "* are motivated to start experimenting with the SQL interface to select subsets of IDC data at different levels of data model (collection, patient, study, series)\n",
         "\n",
-        "If you have any questions about this tutorial, or about searching IDC metadata, please send us an email to support@canceridc.dev or posting your question on [IDC User forum](https://discourse.cancer.dev)!"
+        "If you have any questions about this tutorial, or about searching IDC metadata, please send us an email to support@canceridc.dev or posting your question on [IDC User forum](https://discourse.cancer.dev)!\n",
+        "\n",
+        "This tutorial barely scratches the surface of what you can do with BigQuery SQL. If you are interested in a comprehensive tutorial about BigQuery SQL, check out this [\"Intro to SQL\" course on Kaggle](https://www.kaggle.com/learn/intro-to-sql)!"
       ]
     },
     {

notebooks/getting_started/part3_exploring_cohorts.ipynb

@@ -90,7 +90,7 @@
         "\n",
         "In IDC, a _cohort_ is set of objects stored in IDC that share certain characteristics as defined by metadata.\n",
         "\n",
-        "In the previous tutorial you learned how to use IDC metadata and SQL to filter IDC data and define subsets/cohorts based on such metadata characteristics as cancer location or image modality. You also learned about the hierarchy of data organization in IDC, whereas your \"cohorts\" can be defined at the level of collections, patients, imaging studies or individual images (imaging series).\n",
+        "In the previous tutorial you learned how to use IDC metadata and SQL to filter IDC data and define subsets/cohorts based on such metadata characteristics as cancer location or image modality. You also learned about the hierarchy of data organization in IDC, whereas your \"cohorts\" can be defined at the level of collections, patients, imaging studies, series or individual images (files).\n",
         "\n",
         "In the following cells we will learn:\n",
         "1. How to visualize images from your cohort\n",