ep-nag · Oct 21, 2026 · Oct 21, 2026 · Oct 21, 2026 · Nov 4, 2026 · Nov 6, 2026
@@ -28,6 +28,23 @@ matter of registering different service account credentials, for the different
 projects, to the TKFE portal. The process to create additional service accounts
 is outlined in the following section.
 
+A multi-project setup needs one extra step. Terraform state for every resource
+TKFE provisions is held in a Cloud Storage bucket in the deployment project
+(see [Terraform State](admin_guide.md#terraform-state)), and a service account
+can write the state of the resources it provisions only if it has access to
+that bucket. After creating an additional service account, add it to
+`state_bucket_writers` in
+`tf/state_bucket/deployments/<deployment_name>/admin.auto.tfvars` on the client
+machine and re-apply that directory:
+
+```shell
+terraform -chdir=tf/state_bucket/deployments/<deployment_name> apply
+```
+
+`deploy.sh` never overwrites `admin.auto.tfvars`, so the entry survives
+redeploys. Without this step, provisioning with that credential fails when
+Terraform tries to write state.
+
 ---
 
 ## Service Account Management

@@ -98,6 +98,17 @@ Here are some notes from a developer's perspective:
 - Terraform creates a supporting GCS bucket. This bucket is not only used
   during deployment, but also provides a long-term storage for clusters
   operating within this deployment.
+- Before any of the above, `deploy.sh` applies a second, separate Terraform
+  root, `tf/state_bucket/`, which creates a versioned bucket holding the
+  Terraform state of every resource the FrontEnd goes on to provision. It is a
+  separate root so that tearing the FrontEnd down cannot destroy it
+  (`teardown.sh` removes it only with `--delete-state-bucket`): state for a
+  running cluster must outlive the FrontEnd that created it. `deploy.sh` runs
+  it in `tf/state_bucket/deployments/<deployment_name>/`, which links to the
+  root's `.tf` files and holds that deployment's own local state and
+  variables. The bucket name is passed to the service machine in the
+  `terraform-state-bucket` instance metadata key and ends up as
+  `server.terraform_state_bucket` in `configuration.yaml`.
 - Terraform sets up a Pub/Sub topic for communication between the service
   machine and clusters.
 - Terraform provisions a compute engine virtual machine to be the service
@@ -169,6 +180,22 @@ The directory contains the following:
 Note that life cycles of VPCs and Filestores are managed independently to those
 of clusters.
 
+These run-time directories hold the generated Terraform configuration and the
+log files, but **not** the Terraform state: each directory is configured with a
+GCS backend, so state lives in the state bucket under a prefix made of the
+database's namespace and the directory name
+(`\<namespace>/clusters/cluster_\<id>`, `\<namespace>/vpcs/vpc_\<id>`, and so
+on for `fs`, `images` and `workbenches`), built by
+`cluster_manager.utils.terraform_state_prefix`. The namespace is created with
+the TKFE database (model `StateNamespace`; `python manage.py tfstate_namespace`
+prints it), so a rebuilt server never writes over an earlier database's state.
+Blueprint-generated resources get the backend from `terraform_backend_defaults`
+in the generated blueprint; the raw-template resources (VPCs and workbenches)
+get a `backend.tf`, written into the directory by
+`cluster_manager.utils.write_backend_tf` when the template is copied. Code that
+needs to inspect state uses `cluster_manager.utils.get_terraform_state`, which
+runs `terraform state pull`, rather than opening a local `terraform.tfstate`.
+
 #### For applications
 
 Application data is stored in the shared filesystem `/opt/cluster`. It contains the following sub-directories:
@@ -251,6 +278,7 @@ define the major components:
 | `.../docs/`                 | documentation |
 | `.../infrastructure_files/` | Support files for deploying cloud infrastructure |
 | `.../tf/`                   | Terraform files for deploying the HPC FrontEnd |
+| `.../tf/state_bucket/`      | Separate Terraform root for the Terraform state bucket |
 | `.../website/`              | Source code for the HPC Frontend website |
 
 ##### Infrastructure Files
@@ -347,10 +375,9 @@ Workbenches are created using a template configuration in
 `cluster-toolkit/frontend/infrastructure_files/workbench_tf`. The Terraform
 template was originally based on the Terraform template provided by the
 [Google Cloud Platform Rad-Lab git repo](https://github.com/GoogleCloudPlatform/rad-lab)
-however the configuration diverged during early development. The main reason
-for this divergence was to accommodate running the Jupyter notebook as a
-specific OSLogin user rather than the generic Jupyter user which would make it
-impossible to interact properly with any mounted shared storage.
+however the configuration has since diverged. It now creates a Vertex AI
+Workbench Instance (`google_workbench_instance`) whose post-startup script is
+generated by the FrontEnd, and waits for the instance to become ready.
 
 The process of creating the workbench files is mostly contained within the file
 `cluster-toolkit/frontend/website/ghpcfe/cluster_manager/workbenchinfo.py`. The
@@ -368,7 +395,6 @@ following info gathered by the FrontEnd during the workbench creation process:
 - boot_disk_type
 - boot_disk_size_gb
 - trusted_users
-- image_family
 - owner_id
 - wb_startup_script_name
 - wb_startup_script_bucket
@@ -396,30 +422,34 @@ social ID number set by the owner_id field. It also passes any configured mount
 points into the startup script before the second part of the startup script is
 copied from `infrastructure_files/gcs_bucket/workbench/startup_script_template.sh`.
 
+The script is uploaded to the deployment's GCS bucket and passed to the
+instance in the `post-startup-script` metadata key. On Vertex AI Workbench
+Instances the managed image owns JupyterLab and its systemd unit, so the script
+does not change the Jupyter configuration, its notebook directory or
+`jupyter.service`; doing so stops the instance from becoming ready. JupyterLab
+runs as the image's `jupyter` user, with `/home/jupyter` as the directory shown
+in the web interface.
+
 The startup script runs the following processes when the workbench instance
 boots:
 
-- Query instance metadata for list of users and filter based on the users
-  social ID number to discover the correct format of their OSLogin username.
-- Install nfs-common package via `apt-get`.
-- Make temporary jupyterhome directory in /tmp/ and set user ownership.
-- Make home directory for the user and set user ownership.
-- Copy Jupyter configuration files from `/home/jupyter/.jupyter` to
-  `/tmp/jupyterhome/.jupyter`.
-- Create `DATA_LOSS_WARNING.txt` file with warning message.
-- Configure specified mount points in order specified on FrontEnd.
-- Add symlink to `/tmp/jupyterhome/` which will serve as working directory on
-  the web interface.
-  - This process of mounting and symlinking means the mountpoint will appear
-    in both the jupyter notebook web interface working directory and in the
-    expected location in the root filesystem.
-- Append mount points to `DATA_LOSS_WARNING.txt` file.
-- If `/home` was not mounted as a mount point then create a symlink to
-  `/home` in `/tmp/jupyterhome`.
-- Modify Jupyter config to reflect username and new working directory.
-- Update `/lib/systemd/system/jupyter.service` systemd service file to reflect
-  username and new working directory.
-- Run `systemctl` daemon-reload and restart Jupyter service.
-  - Without updating the Jupyter config and restarting the service then the
-    Jupyter notebook would be running as the jupyter user. This would break
-    permissions used on any mounted shared storage.
+- Query instance metadata for the list of users and filter on the user's
+  social ID number to find their OSLogin username, which is logged to
+  `/tmp/startup.log`.
+- Install the nfs-common package via `apt-get`.
+- If the workbench is attached to a Slurm cluster, install the Slurm client
+  (built from source only if `scontrol` is not already present), copy the
+  cluster's authentication key (`slurm.key`, or `munge.key` for clusters using
+  MUNGE) from the controller, and mount the controller's Slurm configuration
+  at `/mnt/slurm_configs`, linking `slurm.conf` into `/etc/slurm`.
+- Create `/home/jupyter/DATA_LOSS_WARNING.txt` with a warning message.
+- For each mount point, in the order specified on the FrontEnd:
+  - mount the export at its mount path and make it writable by all users
+    (`chmod 777`), since JupyterLab runs as the `jupyter` user;
+  - symlink it into `/home/jupyter`, named after the mount path with `/`
+    replaced by `_` (for example `/data/shared` appears as `data_shared`), so
+    it appears both in the JupyterLab file browser and at its expected
+    location in the root filesystem;
+  - append it to `DATA_LOSS_WARNING.txt`.
+- Log completion to `/tmp/startup.log`. No completion marker is written: the
+  `wait-for-startup` module detects readiness by polling the instance state.
@@ -30,6 +30,7 @@ if [ -z "${SERVER_IP_ADDRESS}" ]; then
 fi
 SERVER_HOSTNAME=$(curl --silent --fail http://metadata/computeMetadata/v1/instance/attributes/hostname -H "Metadata-Flavor: Google")
 config_bucket=$(curl --silent --show-error http://metadata/computeMetadata/v1/instance/attributes/webserver-config-bucket -H "Metadata-Flavor: Google")
+state_bucket=$(curl --silent --show-error http://metadata/computeMetadata/v1/instance/attributes/terraform-state-bucket -H "Metadata-Flavor: Google")
 c2_topic=$(curl --silent --show-error http://metadata/computeMetadata/v1/instance/attributes/ghpcfe-c2-topic -H "Metadata-Flavor: Google")
 deploy_mode=$(curl --silent --show-error http://metadata/computeMetadata/v1/instance/attributes/deploy_mode -H "Metadata-Flavor: Google")
 deployment_name=$(curl --silent --show-error http://metadata/computeMetadata/v1/instance/attributes/deployment_name -H "Metadata-Flavor: Google")
@@ -233,6 +234,7 @@ sudo su - gcluster -c /bin/bash <<EOF
   echo "    host_type: \"GCP\"" >> configuration.yaml
   echo "    gcp_project: \"$GCP_PROJECT\"" >> configuration.yaml
   echo "    gcs_bucket: \"${config_bucket}\"" >> configuration.yaml
+  echo "    terraform_state_bucket: \"${state_bucket}\"" >> configuration.yaml
   echo "    c2_topic: \"${c2_topic}\"" >> configuration.yaml
   echo "    deployment_name: \"${deployment_name}\"" >> configuration.yaml
 

@@ -13,40 +13,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-if [ ! -d /tmp/jupyterhome/home ]; then ln -s /home /tmp/jupyterhome/; fi
-
-echo "modifying jupyter config" | tee -a /tmp/startup.log
-
-cat >>/tmp/jupyterhome/.jupyter/jupyter_notebook_config.py <<+
-jupyter_user = "$USER"
-jupyter_home = "/tmp/jupyterhome"
-sys.path.append(f"{jupyter_home}/.jupyter/")
-c.ServerApp.notebook_dir = "/tmp/jupyterhome"
-+
-
-echo "modifying jupyter service" | tee -a /tmp/startup.log
-cat >/lib/systemd/system/jupyter.service <<+
-[Unit]
-Description=Jupyter Notebook Service
-
-[Service]
-Type=simple
-PIDFile=/run/jupyter.pid
-MemoryHigh=3493718272
-MemoryMax=3543718272
-ExecStart=/bin/bash --login -c '/opt/conda/bin/jupyter lab --config=/tmp/jupyterhome/.jupyter/jupyter_notebook_config.py'
-#User=jupyter
-User=$USER
-Group=$USER
-WorkingDirectory=/tmp/jupyterhome
-Restart=always
-
-[Install]
-WantedBy=multi-user.target
-+
-
-echo "reloading and restarting service" | tee -a /tmp/startup.log
-systemctl daemon-reload
-service jupyter restart
-
-echo BRINGUP COMPLETE
+# Trailer appended to the generated post-startup script.
+# On Vertex AI Workbench Instances the image owns JupyterLab and its
+# systemd unit: do NOT relocate the notebook directory or overwrite the
+# unit, or the instance never becomes ready. Readiness is detected by
+# wait-for-startup polling the instance state, so no completion marker
+# is needed here.
+
+echo "OFE workbench post-startup script finished at $(date)" | tee -a /tmp/startup.log
@@ -18,32 +18,31 @@ limitations under the License.
 | Name | Version |
 | ---- | ------- |
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.12.2 |
-| <a name="requirement_google"></a> [google](#requirement\_google) | >= 3.87.0 |
+| <a name="requirement_google"></a> [google](#requirement\_google) | >= 6.4.0 |
 | <a name="requirement_random"></a> [random](#requirement\_random) | >= 2.0 |
 | <a name="requirement_time"></a> [time](#requirement\_time) | >= 0.7.2 |
 
 ## Providers
 
 | Name | Version |
 | ---- | ------- |
-| <a name="provider_google"></a> [google](#provider\_google) | >= 3.87.0 |
+| <a name="provider_google"></a> [google](#provider\_google) | >= 6.4.0 |
 | <a name="provider_random"></a> [random](#provider\_random) | >= 2.0 |
 | <a name="provider_time"></a> [time](#provider\_time) | >= 0.7.2 |
 
 ## Modules
 
 | Name | Source | Version |
 | ---- | ------ | ------- |
-| <a name="module_project_radlab_ds_analytics"></a> [project\_radlab\_ds\_analytics](#module\_project\_radlab\_ds\_analytics) | terraform-google-modules/project-factory/google | ~> 11.0 |
-| <a name="module_vpc_ai_notebook"></a> [vpc\_ai\_notebook](#module\_vpc\_ai\_notebook) | terraform-google-modules/network/google | ~> 3.0 |
+| <a name="module_project_radlab_ds_analytics"></a> [project\_radlab\_ds\_analytics](#module\_project\_radlab\_ds\_analytics) | terraform-google-modules/project-factory/google | ~> 18.0 |
+| <a name="module_vpc_ai_notebook"></a> [vpc\_ai\_notebook](#module\_vpc\_ai\_notebook) | terraform-google-modules/network/google | ~> 10.0 |
 | <a name="module_waitforstartup"></a> [waitforstartup](#module\_waitforstartup) | ./wait-for-startup | n/a |
 
 ## Resources
 
 | Name | Type |
 | ---- | ---- |
 | [google_compute_instance_iam_member.oslogin_permissions](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_instance_iam_member) | resource |
-| [google_notebooks_instance.ai_notebook](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/notebooks_instance) | resource |
 | [google_project_iam_member.sa_p_notebook_permissions](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/project_iam_member) | resource |
 | [google_project_organization_policy.external_ip_policy](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/project_organization_policy) | resource |
 | [google_project_organization_policy.shielded_vm_policy](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/project_organization_policy) | resource |
@@ -52,6 +51,7 @@ limitations under the License.
 | [google_service_account.sa_p_notebook](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account) | resource |
 | [google_service_account_iam_member.sa_ai_notebook_user_iam](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account_iam_member) | resource |
 | [google_storage_bucket_object.startup_script](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket_object) | resource |
+| [google_workbench_instance.ai_notebook](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/workbench_instance) | resource |
 | [random_id.default](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/id) | resource |
 | [time_sleep.wait_120_seconds](https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/sleep) | resource |
 | [google_compute_network.default](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/compute_network) | data source |
@@ -67,11 +67,11 @@ limitations under the License.
 | <a name="input_boot_disk_type"></a> [boot\_disk\_type](#input\_boot\_disk\_type) | Disk types for notebook instances | `string` | `"PD_SSD"` | no |
 | <a name="input_create_network"></a> [create\_network](#input\_create\_network) | If the module has to be deployed in an existing network, set this variable to false. | `bool` | `false` | no |
 | <a name="input_create_project"></a> [create\_project](#input\_create\_project) | Set to true if the module has to create a project.  If you want to deploy in an existing project, set this variable to false. | `bool` | `false` | no |
+| <a name="input_data_disk_size_gb"></a> [data\_disk\_size\_gb](#input\_data\_disk\_size\_gb) | Size in GB, up to 64000, of the instance's data disk (/home/jupyter). Billed in addition to the boot disk. | `number` | `100` | no |
 | <a name="input_enable_services"></a> [enable\_services](#input\_enable\_services) | Enable the necessary APIs on the project.  When using an existing project, this can be set to false. | `bool` | `false` | no |
 | <a name="input_folder_id"></a> [folder\_id](#input\_folder\_id) | Folder ID where the project should be created. It can be skipped if already setting organization\_id. Leave blank if the project should be created directly underneath the Organization node. | `string` | `""` | no |
 | <a name="input_image_family"></a> [image\_family](#input\_image\_family) | DEPRECATED: Image of the AI notebook. | `string` | `null` | no |
 | <a name="input_image_project"></a> [image\_project](#input\_image\_project) | DEPRECATED: Google Cloud project where the image is hosted. | `string` | `null` | no |
-| <a name="input_instance_image"></a> [instance\_image](#input\_instance\_image) | Image of the AI notebook.<br/><br/>Expected Fields:<br/>name: The name of the image. Mutually exclusive with family.<br/>family: The image family to use. Mutually exclusive with name.<br/>project: The project where the image is hosted. | `map(string)` | <pre>{<br/>  "family": "tf-latest-cpu",<br/>  "project": "deeplearning-platform-release"<br/>}</pre> | no |
 | <a name="input_ip_cidr_range"></a> [ip\_cidr\_range](#input\_ip\_cidr\_range) | Unique IP CIDR Range for AI Notebooks subnet | `string` | `"10.142.190.0/24"` | no |
 | <a name="input_machine_type"></a> [machine\_type](#input\_machine\_type) | Type of VM you would like to spin up | `string` | `"n1-standard-1"` | no |
 | <a name="input_network_name"></a> [network\_name](#input\_network\_name) | Name of the network to be created. | `string` | `"ai-notebook"` | no |