Upgrade r image

.github/workflows/test-terra-jupyter-base.yml

@@ -64,7 +64,7 @@ jobs:
         credentials_json: ${{ secrets.TD_GCP_SA_KEY }}
         create_credentials_file: true
 
-    - name: Set up Cloud SDK
+    - name: Set up Google Cloud CLI
       uses: google-github-actions/setup-gcloud@v3
       with:
         project_id: ${{ secrets.GCP_PROJECT_ID }}
@@ -98,5 +98,6 @@ jobs:
           --volume $GITHUB_WORKSPACE/terra-jupyter-base/tests:/tests \
           --workdir=/tests \
           --entrypoint="" \
+          --env PIP_USER=false \
           terra-jupyter-base:smoke-test \
-          /bin/sh -c "pip3 install pytest; pytest"
+          /bin/bash -c '"$CONDA_ENV_HOME"/bin/python -m pip install pytest && "$CONDA_ENV_HOME"/bin/python -m pytest'

.gitignore

@@ -17,4 +17,7 @@ package-lock.json
 .python_history
 .keras/
 .ammonite/
-.metals/
+.metals/
+# service-account keys fetched from vault at build time
+image-build-account.json
+dspci-wb-gcr-service-account.json

README.md

@@ -5,8 +5,11 @@ This repo provides docker images for running jupyter notebook in [Terra](https:/
 Make sure to go through the [contributing guide](https://github.com/DataBiosphere/terra-docker/blob/master/CONTRIBUTING.md#contributing) as you make changes to this repo.
 
 # Terra Base Images
+
 [terra-base](terra-base/README.md)
 
+[terra-jupyter-base](terra-jupyter-base/README.md)
+
 [terra-jupyter-python](terra-jupyter-python/README.md)
 
 [terra-jupyter-r](terra-jupyter-r/README.md)
@@ -17,6 +20,12 @@ Make sure to go through the [contributing guide](https://github.com/DataBiospher
 
 [terra-jupyter-bioconductor](terra-jupyter-bioconductor/README.md)
 
+## Image dependencies
+
+![Image dependencies](dependencies.png)
+To update this map, install the PlantUML intelliJ extension, then edit the `mind.puml` file.
+
+
 # How to create your own Custom image to use with notebooks on Terra
 Custom docker images need to use a Terra base image (see above) in order to work with the service that runs notebooks on Terra.
 * You can use any of the base images above
@@ -83,12 +92,27 @@ Detailed documentation on how to integrate the terra-docker image with Leonardo
 - Ensure that the `terra-docker-versions-candidate.json` file (which is what the UI sources the dropdown from) in the `terra-docjker-image-documentation-[env]` bucket correclty references your new docker image
 - [Update the terra-docker version candidate json](https://broadworkbench.atlassian.net/wiki/spaces/IA/pages/2519564289/Integrating+new+Terra+docker+images+with+Leonardo#6.-Update-terra-docker-versions-candidate.json)
 
+## UV and updating dependencies
+This repo uses [uv](https://pypi.org/project/uv/) to manage python dependencies. 
+The uv.lock file at the root of the directory ensures package versions across all the docker images are compatible. 
+Each image has a pyproject.toml file that defines the additional dependencies for that image.
+- To make a general update to all dependencies, cd into the image directory and run `uv update`.
+- If you want to update a specific dependency, you can run `uv add [package name]@[version]` to update to a specific version or just `uv add [package name]` to update to the latest version.
+- Because the uv.lock file is at the root of the repo, you will need to run `uv sync` in each image directory after making changes to the dependencies to update the lockfile.
+
+IMPORTANT NOTE: The terra-base image has a separate uv.lock file because its pyproject.toml defines dependencies for the isolated jupyter server environment, which has different dependencies than the other images. 
+This means that if you are updating dependencies for the terra-base image, you will need to cd into the terra-base directory to update the uv.lock file for that image. 
+For all other images, the uv.lock file at the root of the repo is used and therefore docker image must be built from the root of the repo.
+
 ## Testing your image manually
 
-Build the image:
-run `docker build [your_dir] -t [name]`.
+#### Build the image from the root of the repo:
+This is necessary to ensure the workspace `uv.lock` file can be accessed. 
+The one exception is if you are building the `terra-base` image, you will need to cd into the `terra-base` directory and build from there)
+
+run `docker build -f [your_dir]/Dockerfile . -t [name]`.
 
-`docker build terra-base -t terra-base`
+`docker build -f terra-jupyter-base/Dockerfile . -t  us.gcr.io/broad-dsp-gcr-public/terra-base:{tag}`
 
 If you're on an M1 and building an image from a locally built image, replace the current FROM command:
 
@@ -111,7 +135,8 @@ You can gain root access and open a bash terminal as follows:
 docker run --rm -it -u root -p 8000:8000 --entrypoint /bin/bash us.gcr.io/broad-dsp-gcr-public/terra-jupyter-base:0.0.7
 ```
 
-Running locally is conventient for quick development and exploring the image. However it has some limitations compared to running through Terra. Namely:
+Running locally is convenient for quick development and exploring the image. 
+However it has some limitations compared to running through Terra. Namely:
 - there are no service account credentials when run locally
 - there are no environment variables like `GOOGLE_PROJECT`, `WORKSPACE_NAME`, `WORKSPACE_BUCKET`, etc when running locally
 - there is no workspace-syncing when run locally
@@ -168,10 +193,6 @@ The scripts folder has scripts used for building.
 - `generate_package_docs.py` This script is run once by build.sh each time an image is built. It is used to generate a .json with the versions for the packages in the image.
 - `generate_version_docs.py` This script is run each time an image is built. It builds a new file master version file for the UI to look up the current versions to reference.
 
-## Image dependencies
-
-Note that this dependency graph needs to be updated!
-![Image dependencies](dependencies.png)
 
 ## Push Images to GCR
 

build.sh

@@ -1,10 +1,17 @@
 #!/bin/bash
 # Run `gcloud auth configure-docker --quiet` before running this script
 # Example: ./build.sh terra-jupyter-base
+# Example: ./build.sh terra-jupyter-base true
+# Example: KEEP_LOCAL_IMAGE=true ./build.sh terra-jupyter-base true
+#
+# Optional environment variables:
+#   KEEP_LOCAL_IMAGE  true keeps the pulled image in the local docker store after the build
 set -e -x
+set -o pipefail
 
 IMAGE_DIR=$1
-SKIP_GCP_AUTH=$2
+SKIP_GCP_AUTH=${2:-${SKIP_GCP_AUTH:-}}
+KEEP_LOCAL_IMAGE=${KEEP_LOCAL_IMAGE:-false}
 VERSION=$(cat config/conf.json | jq -r ".image_data | .[] | select(.name == \"$IMAGE_DIR\") | .version")
 
 TAG_NAME=$(git log --pretty=format:'%h' -n 1)
@@ -26,36 +33,84 @@ if [[ $VAULT_LOCATION == *"jenkins"* ]]; then
     VAULT_LOCATION="/etc/vault-token-dsde"
 fi
 
-if [[ -z "${SKIP_GCP_AUTH}" ]] || [["${SKIP_GCP_AUTH}" != "true"]]; then
+if [[ "${SKIP_GCP_AUTH}" != "true" ]]; then
+  # Docker silently creates the mount source as a directory when it does not
+  # exist, which makes vault fail with "is a directory" instead of anything
+  # useful. Check up front so the error names the real problem.
+  if [[ ! -f "$VAULT_LOCATION" ]]; then
+      echo "No vault token file at $VAULT_LOCATION (it must be a file, not a directory)."
+      echo "Create one with:"
+      echo "  docker run -it --rm -v \$HOME:/root -e VAULT_ADDR=https://clotho.broadinstitute.org:8200 \\"
+      echo "    broadinstitute/dsde-toolbox:latest vault login -method=github"
+      echo "Or re-run with SKIP_GCP_AUTH=true to use your existing gcloud credentials."
+      #unreserved exit code for checking in jenkins
+      exit 16
+  fi
+
   # will fail if you are not gcloud authed as dspci-wb-gcr-service-account or secret/dsde/firecloud/common/image-build-account.json
   # the below works locally but not on jenkins for some reason, using a different account until a resolution is found
   # docker run --rm  -v $VAULT_LOCATION:/root/.vault-token:ro broadinstitute/dsde-toolbox:latest vault read --format=json secret/dsde/dsp-techops/common/dspci-wb-gcr-service-account.json | jq .data > dspci-wb-gcr-service-account.json
   # gcloud auth activate-service-account --key-file=dspci-wb-gcr-service-account.json
   docker run --rm  -v $VAULT_LOCATION:/root/.vault-token:ro broadinstitute/dsde-toolbox:latest vault read --format=json secret/dsde/firecloud/common/image-build-account.json | jq .data > image-build-account.json
+
+  # jq happily writes an empty file when the vault read produced nothing, which
+  # surfaces later as a confusing "Could not read json file" from gcloud.
+  if [[ ! -s image-build-account.json ]]; then
+      echo "vault read produced an empty image-build-account.json; see the error above."
+      #unreserved exit code for checking in jenkins
+      exit 16
+  fi
+
   gcloud auth activate-service-account --key-file=image-build-account.json
   gcloud auth configure-docker --quiet
 fi
 
 
-docker image build ./$IMAGE_DIR \
+# Build and push in a single buildx step.
+#
+# Leonardo's registry client only accepts Docker schema 2 media types, so an OCI
+# manifest or index makes its image auto-detection fail with MANIFEST_UNKNOWN. A
+# plain `docker build && docker push` emits an OCI index whenever the daemon uses
+# the containerd image store, which is the default in recent Docker versions.
+#   --platform      pins the target arch; otherwise the manifest records the
+#                   builder host's arch even when the FROM line is pinned to amd64
+#   --provenance    attestations are published as an extra manifest, which turns
+#   --sbom          the push into an index even for a single-platform image
+#   oci-mediatypes  false forces application/vnd.docker.distribution.manifest.v2+json
+docker buildx build ./$IMAGE_DIR \
+  --platform linux/amd64 \
+  --provenance=false \
+  --sbom=false \
+  --output type=image,oci-mediatypes=false,push=true \
   --tag $GCR_IMAGE_REPO/$IMAGE_DIR:$TAG_NAME \
   --tag $GCR_IMAGE_REPO/$IMAGE_DIR:$VERSION \
   --tag $GCR_IMAGE_REPO/$IMAGE_DIR:latest
 
-docker push $GCR_IMAGE_REPO/$IMAGE_DIR:$TAG_NAME
-docker push $GCR_IMAGE_REPO/$IMAGE_DIR:$VERSION
-docker push $GCR_IMAGE_REPO/$IMAGE_DIR:latest
+# Fail the build rather than publishing something Leonardo cannot read.
+MANIFEST_TYPE=$(docker buildx imagetools inspect --raw $GCR_IMAGE_REPO/$IMAGE_DIR:$VERSION | jq -r '.mediaType')
+if [ "$MANIFEST_TYPE" != "application/vnd.docker.distribution.manifest.v2+json" ]; then
+    echo "Pushed manifest is $MANIFEST_TYPE, expected application/vnd.docker.distribution.manifest.v2+json."
+    echo "Leonardo will not be able to auto-detect this image."
+    #unreserved exit code for checking in jenkins
+    exit 15
+fi
+
+# buildx pushed straight to the registry without loading the image into the local
+# store, so pull it back for the doc-generation container below.
+docker pull $GCR_IMAGE_REPO/$IMAGE_DIR:$VERSION
 
 docker kill $IMAGE_DIR || true
 docker rm -f $IMAGE_DIR || true
 docker run --rm -itd -u root -e PIP_USER=false --entrypoint='/bin/bash' --name $IMAGE_DIR $GCR_IMAGE_REPO/$IMAGE_DIR:$VERSION
 
 gcloud auth list
-python scripts/generate_package_docs.py "$IMAGE_DIR"
+python3 scripts/generate_package_docs.py "$IMAGE_DIR"
 
 docker kill $IMAGE_DIR || true
 docker rm -f $IMAGE_DIR || true
-docker image rm -f $GCR_IMAGE_REPO/$IMAGE_DIR:$VERSION
-docker image rm -f $GCR_IMAGE_REPO/$IMAGE_DIR:$TAG_NAME
+if [[ "${KEEP_LOCAL_IMAGE}" != "true" ]]; then
+  docker image rm -f $GCR_IMAGE_REPO/$IMAGE_DIR:$VERSION
+  docker image rm -f $GCR_IMAGE_REPO/$IMAGE_DIR:$TAG_NAME
+fi
 
 echo "Successfully completed build script for $IMAGE_DIR"

build_smoke_test_image.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
 # Perform a local build of an image for smoke-testing purposes. Also build its tree of base images, when missing.
 #
-# Example: ./build_smoke_test_image.sh terra-jupyter-python  
+# Example: ./build_smoke_test_image.sh terra-jupyter-base
 #
 # Current working directory must be the directory in which this script resides (one level above all the Dockerfiles).
 # It walks down each relevant image directory, making changes to the FROM statements,
@@ -13,9 +13,10 @@ set -o nounset
 set -o xtrace 
 
 build_smoke_test_image() {
-  local IMAGE_TYPE=$1
+  local IMAGE_TYPE=${1:-"terra-jupyter-base"}
   pushd ${IMAGE_TYPE}
-  local BASE_IMAGES=$( egrep '^FROM (\S+)' Dockerfile |tr -s ' ' | cut -d ' ' -f 2 )
+  # The image is the first FROM argument that is not an option such as --platform=${BASE_PLATFORM}
+  local BASE_IMAGES=$( awk '/^FROM/ { for (i=2; i<=NF; i++) if ($i !~ /^--/) { print $i; break } }' Dockerfile )
 
   local BASE_IMAGE
   for BASE_IMAGE in ${BASE_IMAGES}; do
@@ -43,4 +44,4 @@ build_smoke_test_image() {
 # Log available disk space prior to building image
 df -kh
 
-build_smoke_test_image $1
+build_smoke_test_image "${1:-}"

config/conf.json

@@ -30,7 +30,7 @@
             "automated_flags" : {
                 "generate_docs" : true,
                 "include_in_ui" : true,
-                "build" : true,
+                "build" : false,
                 "include_in_custom_dataproc" : true,
                 "include_in_custom_gce" : true
             }
@@ -51,7 +51,7 @@
             "automated_flags" : {
                 "generate_docs" : true,
                 "include_in_ui" : true,
-                "build" : true,
+                "build" : false,
                 "include_in_custom_dataproc" : true,
                 "include_in_custom_gce" : false
             }
@@ -72,7 +72,7 @@
             "automated_flags" : {
                 "generate_docs" : true,
                 "include_in_ui" : true,
-                "build" : true,
+                "build" : false,
                 "include_in_custom_dataproc" : true,
                 "include_in_custom_gce" : true
             }
@@ -86,10 +86,10 @@
             "packages" : {
 
             },
-            "version" : "1.1.4",
+            "version" : "2.0.1",
             "automated_flags" : {
                 "generate_docs" : true,
-                "include_in_ui" : false,
+                "include_in_ui" : true,
                 "build" : true,
                 "include_in_custom_dataproc" : true,
                 "include_in_custom_gce" : true
@@ -108,7 +108,7 @@
             "automated_flags" : {
                 "generate_docs" : true,
                 "include_in_ui" : true,
-                "build" : true,
+                "build" : false,
                 "include_in_custom_dataproc" : true,
                 "include_in_custom_gce" : true
             }
@@ -122,9 +122,9 @@
             "packages" : {
 
             },
-            "version" : "2.2.7",
+            "version" : "3.0.0",
             "automated_flags" : {
-                "include_in_ui" : false,
+                "include_in_ui" : true,
                 "generate_docs" : true,
                 "build" : true,
                 "include_in_custom_dataproc" : true,
@@ -146,7 +146,7 @@
             "automated_flags" : {
                 "include_in_ui" : true,
                 "generate_docs" : true,
-                "build" : true,
+                "build" : false,
                 "include_in_custom_dataproc" : true,
                 "include_in_custom_gce" : true
             }

mind.puml

@@ -1,9 +1,8 @@
 @startmindmap
 * terra-jupyter-base
 ** terra-jupyter-python
-*** terra-jupyter-gatk
 *** terra-jupyter-hail
 ** terra-jupyter-r
-*** terra-jupyter-gatk
 *** terra-jupyter-bioconductor
+*** terra-jupyter-gatk
 @endmindmap

pyproject.toml

@@ -0,0 +1,20 @@
+[project]
+name = "terra-docker"
+version = "2.0.0"
+description = "Collection of inter-dependent docker images for Terra notebooks and services"
+readme = "README.md"
+requires-python = ">=3.13, <3.14"
+dependencies = [
+    "setuptools<71",
+    "terra-jupyter-base",
+    "terra-jupyter-python"
+]
+
+[tool.uv.workspace]
+# NOTE: We explicitly exclude terra-base from this because that defines the dependencies
+# for the isolated jupyter server NOT the conda environment for the notebooks
+members = ["terra-jupyter-base", "terra-jupyter-python"]
+
+[tool.uv.sources]
+terra-jupyter-base = { workspace = true }
+terra-jupyter-python = { workspace = true }

terra-jupyter-base/CHANGELOG.md

@@ -1,3 +1,27 @@
+## 2.0.1 - 2026-10-16
+- Fix vulnerabilities reported in 2.0.0
+  - Conda base env: upgrade `anyio` (>= 4.14.2), `pyjwt` (>= 2.14.0) and `cryptography` (>= 50.0.0)
+  - Drop `python3-openssl`, which pulled in Ubuntu's `python3-cryptography` 46.0.5
+  - Remove the unused `/usr/bin/pebble` (Go 1.26.7) shipped by the `ubuntu:26.04` base
+  - google-cloud-cli: upgrade its bundled `cryptography` (>= 50.0.0) and `pyopenssl` (>= 26.4.0), and remove the
+    optional `gcloud-crc32c` binary (Go 1.26.6); `gcloud storage` falls back to the bundled `google_crc32c`
+  - npm: replace its bundled `brace-expansion` (5.0.12) and `undici` (6.29.1)
+  - nbclassic: replace its bundled `underscore` with 1.13.8
+
+Image URL: `us.gcr.io/broad-dsp-gcr-public/terra-jupyter-base:2.0.1`
+
+## 2.0.0 - 2026-10-16
+- Build off Ubuntu 26.04 with base image `nvidia/cuda:13.4.2-base-ubuntu26.04` (merged with terra-base)
+- Python 3.13 (Miniconda `py313_26.7.1-1`) for conda base, the Jupyter server venv and the user's conda env
+  - Downgraded from 3.14: Tensorflow and several scientific packages used by the child images have no Python 3.14 wheels yet
+  - Ubuntu's `/usr/bin/python3` (3.14) is not used for notebooks; `python3-pip` is no longer installed
+- Jupyter server runs in an isolated uv venv at `/etc/jupyter` (`pyproject.toml` / `uv.lock`), Notebook 7 + nbclassic
+- User's conda env at `/opt/conda/envs/python3`, registered as the `python3` kernel in `/usr/local/share/jupyter/kernels`
+- R and the CRAN `resolute-cran40` apt repository; the R kernel is registered by terra-jupyter-r
+- google-cloud-cli and OpenJDK 25
+
+Image URL: `us.gcr.io/broad-dsp-gcr-public/terra-jupyter-base:2.0.0`
+
 ## 1.1.4 - 2025-04-24
 - Build off Ubuntu 22 with base image `gcr.io/deeplearning-platform-release/tf2-cu123.2-17.py310`