diff --git a/README.md b/README.md index 1c66db11..42184e08 100644 --- a/README.md +++ b/README.md @@ -62,9 +62,11 @@ The folder `examples` contains the following Terraform implementation examples : | AWS | [aws-databricks-uc-bootstrap](examples/aws-databricks-uc-bootstrap/) | AWS UC | | AWS | [aws-remote-backend-infra](examples/aws-remote-backend-infra/) | Simple example on remote backend | | AWS | [aws-workspace-config](examples/aws-workspace-config/) | Configure workspace objects | -| GCP | [gcp-sa-provisionning](examples/gcp-sa-provisionning/) | Provisionning of the identity with the permissions required to deploy on GCP. | -| GCP | [gcp-basic](examples/gcp-basic/) | Workspace Deployment with managed vpc | -| GCP | [gcp-byovpc](examples/gcp-byovpc/) | Workspace Deployment with customer-managed vpc | +| GCP | [gcp-sa-provisioning](examples/gcp-sa-provisioning/) | Provisioning the identity (service account) with permissions required to deploy on GCP | +| GCP | [gcp-basic](examples/gcp-basic/) | Workspace deployment with Databricks-managed VPC | +| GCP | [gcp-byovpc](examples/gcp-byovpc/) | Workspace deployment with customer-managed VPC (Terraform creates the VPC) | +| GCP | [gcp-existing-vpc](examples/gcp-existing-vpc/) | Workspace deployment into a pre-existing VPC | +| GCP | [gcp-with-psc-exfiltration-protection](examples/gcp-with-psc-exfiltration-protection/) | Workspace with PrivateLink (PSC), private DNS, and restricted egress (hub-and-spoke topology) | ### Modules The folder `modules` contains the following Terraform modules : @@ -72,6 +74,7 @@ The folder `modules` contains the following Terraform modules : | Cloud | Module | Description | | ----- |-----------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | All | [databricks-department-clusters](modules/databricks-department-clusters/) | Terraform module that creates Databricks resources for a team | +| All | [databricks/security-settings](modules/databricks/security-settings/) | Cloud-neutral workspace security settings: Compliance Security Profile, Enhanced Security Monitoring, automatic cluster update, and IP access lists | | Azure | [adb-lakehouse](modules/adb-lakehouse/) | Lakehouse terraform blueprints | | Azure | [adb-lakehouse-uc](modules/adb-lakehouse-uc/) | Provisioning Unity Catalog resources and accounts principals | | Azure | [adb-with-private-link-standard](modules/adb-with-private-link-standard/) | Provisioning Databricks on Azure with Private Link - Standard deployment | @@ -89,9 +92,13 @@ The folder `modules` contains the following Terraform modules : | AWS | [aws-workspace-with-firewall](modules/aws-workspace-with-firewall/) | Provisioning AWS Databricks E2 with an AWS Firewall | | AWS | [aws-exfiltration-protection](modules/aws-exfiltration-protection/) | An implementation of [Data Exfiltration Protection on AWS](https://www.databricks.com/blog/2021/02/02/data-exfiltration-protection-with-databricks-on-aws.html) | | AWS | aws-workspace-with-private-link | Coming soon | -| GCP | [gcp-sa-provisionning](modules/gcp-sa-provisionning/) | Provisions the identity (SA) with the correct permissions | -| GCP | [gcp-workspace-basic](modules/gcp-workspace-basic/) | Provisions a workspace with managed VPC | -| GCP | [gcp-workspace-byovpc](modules/gcp-workspace-byovpc/) | Workspace with customer-managed VPC. | +| GCP | [gcp/databricks-workspace](modules/gcp/databricks-workspace/) | Composer that orchestrates network, PSC, workspace, and DNS submodules based on scenario flags | +| GCP | [gcp/network](modules/gcp/network/) | VPC, subnet, router, NAT, peering, and shared-VPC binding (create or data-source lookup) | +| GCP | [gcp/private-connectivity](modules/gcp/private-connectivity/) | PSC endpoints (frontend, backend, hub-transit) and restricted-egress firewall rules | +| GCP | [gcp/workspace](modules/gcp/workspace/) | All databricks_mws_* resources: networks, workspaces, vpc_endpoint, private_access_settings | +| GCP | [gcp/dns](modules/gcp/dns/) | Private DNS zones (gcp.databricks.com, gcr.io, googleapis.com, pkg.dev) for restricted-egress workspaces | +| GCP | [gcp/service-account](modules/gcp/service-account/) | Service account with the IAM permissions required to provision Databricks workspaces | +| GCP | [gcp/unity-catalog](modules/gcp/unity-catalog/) | Metastore, GCS bucket, storage credential, external location, and default catalog | ### CI/CD pipelines The `cicd-pipelines` folder contains the following implementation examples of pipeline: diff --git a/docs/cross-cloud-module-contract.md b/docs/cross-cloud-module-contract.md new file mode 100644 index 00000000..61527d84 --- /dev/null +++ b/docs/cross-cloud-module-contract.md @@ -0,0 +1,86 @@ +# Cross-Cloud Module Contract + +Normative conventions for `modules//` in this repository. The GCP tree +is the reference implementation (PR #233). Azure and AWS refactors are +reviewed against this document. + +## 1. Module slots + +Every cloud provides these modules, defined by responsibility: + +| Slot | Responsibility | GCP | Azure (dialect) | AWS (dialect) | +|---|---|---|---|---| +| `network` | Data-plane network: create or look up VPC/VNet, subnets, NAT, hub-spoke peering, shared-network bindings | `modules/gcp/network` | VNet + subnets + NSGs | VPC + subnets + NAT | +| `private-connectivity` | Private endpoints to the Databricks control plane + egress firewall stack | PSC endpoints | Private Endpoints | VPC endpoints (PrivateLink) | +| `workspace` | Everything registering the workspace with the Databricks control plane | `databricks_mws_*` | `azurerm_databricks_workspace` + access connector | `databricks_mws_*` | +| `dns` | Private DNS zones/records for the private endpoints | Cloud DNS private + peering zones | Private DNS zones | Route 53 private hosted zones | +| `databricks-workspace` | Composer: takes scenario flags, conditionally instantiates the slots (`network → private-connectivity → workspace → dns`) | done | future | future | +| `serverless-connectivity` (reserved) | NCC: serverless private connectivity to customer resources | pending GCP release (near Public Preview, Jul 2026) | `databricks_mws_network_connectivity_config` | same | + +Cloud-neutral modules (identical resources on every cloud) live under +`modules/databricks/` (first resident: `security-settings`). + +## 2. Frozen interface (identical names on every cloud) + +Composer variables: `prefix`, `workspace_name`, `databricks_account_id`, +`tags`; `private_link_frontend`, `private_link_backend`, +`private_access_only`, `restricted_egress`; `serverless_egress_mode` +(`unmanaged|full|restricted`), `serverless_allowed_internet_destinations`, +`serverless_allowed_storage_destinations`, `serverless_egress_enforcement` +(`enforced|dry_run`); `cmek_managed_services_key_id`, `cmek_storage_key_id` +(the key-reference format is per-cloud). + +The network-source variable uses the cloud's noun (`vpc_source` / +`vnet_source`) and is an object with frozen keys and values: `{ spoke = +databricks_managed | create | existing, hub = create | existing (optional; +only settable when spoke is customer-managed) }` — `hub` is consumed only by +egress-restricted topologies. The boolean `enable_hub_spoke_peering` (default +true) is frozen alongside it. + +Composer outputs: `workspace_id`, `workspace_url`, `suffix`, +`serverless_network_policy_id`, plus per-cloud network outputs following the +noun table. + +Cloud dialect constraints are allowed where the platform demands them and +must be enforced as preconditions with an explanatory error message (GCP: +`private_link_frontend == private_link_backend`, because +`mws_networks.vpc_endpoints` requires both endpoint references). + +## 3. Two-tier naming + +Tier 1 (frozen): everything in section 2, plus the vocabulary `hub`/`spoke` +and `frontend`/`backend`. + +Tier 2 (native nouns): each cloud names cloud objects with its own noun. + +| Concept | GCP | Azure | AWS | +|---|---|---|---| +| Network | `*_vpc_*` | `*_vnet_*` | `*_vpc_*` | +| Private endpoint mechanism | `psc` | `private_endpoint` | `vpc_endpoint` | +| Region | `google_region` | `azure_region` | `aws_region` | +| Resource container | `google_project` | `azure_resource_group` | (account-level, none) | +| Endpoint subnet | `psc_subnet_cidr` | `private_endpoint_subnet_cidr` | `endpoint_subnet_cidrs` | +| Endpoint IP outputs | `frontend_psc_ip_` | `frontend_endpoint_ip_` | n/a (ENI-based) | + +## 4. File shape + +Modules: one concern per `.tf` file; `versions.tf` declares +`required_version` + `required_providers` floors only (`google >= 6.0`, +`databricks >= 1.81.1`, `random >= 3.0`); no `provider {}` blocks. +Examples: `versions.tf` (pessimistic pins `~> 6.17` / `~> 1.81`) + +`providers.tf` + `main.tf` + `variables.tf` + `outputs.tf` + +`terraform.tfvars` + `README.md` + `Makefile`. + +Every variable and output has a `description`. Module READMEs carry a +`## Usage` HCL block above the terraform-docs markers. + +## 5. Validation & testing standard + +- Cross-variable rules live in the composer's `preconditions.tf` on a + `terraform_data` resource; the PR description documents the rule table. +- `count`/`for_each` must never depend on apply-time values. +- Each composer scenario has a positive fixture under `tests//` + that completes `terraform plan` offline; preconditions are covered by + `tests/negative-*` fixtures that fail plan with the expected message (at + minimum, every rule that guards a scenario flag combination). +- Submodules carry per-scenario plan fixtures. diff --git a/examples/gcp-basic/README.md b/examples/gcp-basic/README.md index 894b51e4..35ce2983 100644 --- a/examples/gcp-basic/README.md +++ b/examples/gcp-basic/README.md @@ -1,30 +1,36 @@ -# Provisioning Databricks workspace on GCP with managed VPC -========================= +# examples/gcp-basic — Databricks-managed VPC -In this template, we show how to deploy a workspace with managed VPC. +Calls `modules/gcp/databricks-workspace` with `vpc_source = { spoke = "databricks_managed" }`. +The Databricks platform provisions the workspace VPC; you provide only the GCP +project, region, and prefix. +## Prerequisites -## Requirements - -- You need to have run gcp-sa-provisionning and have a service account to fill in the variables. -- If you want to deploy to a new project, you will need to grant the custom role generated in that template to the service acount in the new project. -- The Service Account needs to be added as Databricks Admin in the account console - -## Run as an SA +- A GCP project with the Databricks platform onboarded +- A service account with workspace-creator role (see `examples/gcp-sa-provisioning`) +- Databricks account ID -You can do the same thing by provisionning a service account that will have the same permissions - and associate the key associated to it. +## Apply +```bash +terraform init +terraform apply +``` -## Run the tempalte +## Migrating from the old example -- You need to fill in the `variables.tf` -- run `terraform init` -- run `teraform apply` +This example previously called `modules/gcp-workspace-basic`. State from the +old apply does **not** migrate cleanly to the new composer because the +`databricks_mws_workspaces` resource address differs. Re-apply on clean state. ## Requirements -No requirements. +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.5 | +| [databricks](#requirement\_databricks) | ~> 1.81 | +| [google](#requirement\_google) | ~> 6.17 | ## Providers @@ -34,7 +40,7 @@ No providers. | Name | Source | Version | |------|--------|---------| -| [gcp-basic](#module\_gcp-basic) | github.com/databricks/terraform-databricks-examples/modules/gcp-workspace-basic | n/a | +| [workspace](#module\_workspace) | ../../modules/gcp/databricks-workspace | n/a | ## Resources @@ -45,18 +51,17 @@ No resources. | Name | Description | Type | Default | Required | |------|-------------|------|---------|:--------:| | [databricks\_account\_id](#input\_databricks\_account\_id) | Databricks Account ID | `string` | n/a | yes | -| [databricks\_google\_service\_account](#input\_databricks\_google\_service\_account) | Email of the service account used for deployment | `string` | n/a | yes | -| [delegate\_from](#input\_delegate\_from) | Identities to allow to impersonate created service account (in form of user:user.name@example.com, group:deployers@example.com or serviceAccount:sa1@project.iam.gserviceaccount.com) | `list(string)` | n/a | yes | -| [google\_project](#input\_google\_project) | Google project for VCP/workspace deployment | `string` | n/a | yes | -| [google\_region](#input\_google\_region) | Google region for VCP/workspace deployment | `string` | n/a | yes | -| [google\_zone](#input\_google\_zone) | Zone in GCP region | `string` | n/a | yes | -| [prefix](#input\_prefix) | Prefix to use in generated VPC name | `string` | n/a | yes | -| [workspace\_name](#input\_workspace\_name) | Name of the workspace to create | `string` | n/a | yes | +| [databricks\_google\_service\_account](#input\_databricks\_google\_service\_account) | Service account email used for Databricks provider authentication | `string` | n/a | yes | +| [google\_project](#input\_google\_project) | GCP project where the workspace will be created | `string` | n/a | yes | +| [google\_region](#input\_google\_region) | GCP region for workspace deployment | `string` | n/a | yes | +| [google\_zone](#input\_google\_zone) | GCP zone (used by the google provider) | `string` | n/a | yes | +| [prefix](#input\_prefix) | Prefix used to name generated resources | `string` | n/a | yes | +| [workspace\_name](#input\_workspace\_name) | Workspace name | `string` | n/a | yes | ## Outputs | Name | Description | |------|-------------| -| [databricks\_host](#output\_databricks\_host) | n/a | -| [databricks\_token](#output\_databricks\_token) | n/a | +| [workspace\_id](#output\_workspace\_id) | Databricks workspace ID | +| [workspace\_url](#output\_workspace\_url) | Databricks workspace URL | diff --git a/examples/gcp-basic/main.tf b/examples/gcp-basic/main.tf index 372bbf9f..6bf18031 100644 --- a/examples/gcp-basic/main.tf +++ b/examples/gcp-basic/main.tf @@ -1,9 +1,11 @@ -module "gcp-basic" { - source = "github.com/databricks/terraform-databricks-examples/modules/gcp-workspace-basic" +module "workspace" { + source = "../../modules/gcp/databricks-workspace" + + prefix = var.prefix databricks_account_id = var.databricks_account_id google_project = var.google_project google_region = var.google_region - prefix = var.prefix workspace_name = var.workspace_name - delegate_from = var.delegate_from + + vpc_source = { spoke = "databricks_managed" } } diff --git a/examples/gcp-basic/outputs.tf b/examples/gcp-basic/outputs.tf index d6b170a9..81a92ab3 100644 --- a/examples/gcp-basic/outputs.tf +++ b/examples/gcp-basic/outputs.tf @@ -1,9 +1,9 @@ - -output "databricks_host" { - value = databricks_mws_workspaces.databricks_workspace.workspace_url +output "workspace_id" { + value = module.workspace.workspace_id + description = "Databricks workspace ID" } -output "databricks_token" { - value = databricks_mws_workspaces.databricks_workspace.token[0].token_value - sensitive = true +output "workspace_url" { + value = module.workspace.workspace_url + description = "Databricks workspace URL" } diff --git a/examples/gcp-basic/init.tf b/examples/gcp-basic/providers.tf similarity index 66% rename from examples/gcp-basic/init.tf rename to examples/gcp-basic/providers.tf index a8ea9f9f..edead550 100644 --- a/examples/gcp-basic/init.tf +++ b/examples/gcp-basic/providers.tf @@ -1,14 +1,3 @@ -terraform { - required_providers { - databricks = { - source = "databricks/databricks" - } - google = { - source = "hashicorp/google" - } - } -} - provider "google" { project = var.google_project region = var.google_region @@ -19,4 +8,4 @@ provider "databricks" { host = "https://accounts.gcp.databricks.com" google_service_account = var.databricks_google_service_account account_id = var.databricks_account_id -} \ No newline at end of file +} diff --git a/examples/gcp-basic/terraform.tfvars b/examples/gcp-basic/terraform.tfvars new file mode 100644 index 00000000..8405cca5 --- /dev/null +++ b/examples/gcp-basic/terraform.tfvars @@ -0,0 +1,7 @@ +databricks_account_id = "" +databricks_google_service_account = "" +google_project = "" +google_region = "" +google_zone = "" +prefix = "" +workspace_name = "" diff --git a/examples/gcp-basic/variables.tf b/examples/gcp-basic/variables.tf index 9805c04b..4b02c043 100644 --- a/examples/gcp-basic/variables.tf +++ b/examples/gcp-basic/variables.tf @@ -4,38 +4,32 @@ variable "databricks_account_id" { } variable "databricks_google_service_account" { - description = "Email of the service account used for deployment" type = string + description = "Service account email used for Databricks provider authentication" } variable "google_project" { type = string - description = "Google project for VCP/workspace deployment" + description = "GCP project where the workspace will be created" } variable "google_region" { type = string - description = "Google region for VCP/workspace deployment" + description = "GCP region for workspace deployment" } variable "google_zone" { - description = "Zone in GCP region" type = string + description = "GCP zone (used by the google provider)" } variable "prefix" { type = string - description = "Prefix to use in generated VPC name" + description = "Prefix used to name generated resources" } variable "workspace_name" { - description = "Name of the workspace to create" type = string + description = "Workspace name" } -variable "delegate_from" { - description = "Identities to allow to impersonate created service account (in form of user:user.name@example.com, group:deployers@example.com or serviceAccount:sa1@project.iam.gserviceaccount.com)" - type = list(string) -} - - diff --git a/examples/gcp-with-psc-exfiltration-protection/terraform.tf b/examples/gcp-basic/versions.tf similarity index 59% rename from examples/gcp-with-psc-exfiltration-protection/terraform.tf rename to examples/gcp-basic/versions.tf index 791cc0c9..150d0ab5 100644 --- a/examples/gcp-with-psc-exfiltration-protection/terraform.tf +++ b/examples/gcp-basic/versions.tf @@ -1,15 +1,13 @@ terraform { + required_version = ">= 1.5" required_providers { databricks = { source = "databricks/databricks" - version = ">=1.81.1" + version = "~> 1.81" } google = { source = "hashicorp/google" - version = "6.17.0" - } - random = { - source = "hashicorp/random" + version = "~> 6.17" } } -} \ No newline at end of file +} diff --git a/examples/gcp-byovpc/README.md b/examples/gcp-byovpc/README.md index 8dc13eba..ce5af809 100644 --- a/examples/gcp-byovpc/README.md +++ b/examples/gcp-byovpc/README.md @@ -1,72 +1,83 @@ -# Provisioning Databricks workspace on GCP with a custom VPC -========================= +# examples/gcp-byovpc — Customer-managed VPC -In this template, we show how to deploy a workspace with a custom vpc. +Calls `modules/gcp/databricks-workspace` with `vpc_source = { spoke = "create" }`. Terraform +creates the spoke VPC + subnet + Cloud Router + NAT, then registers the network +with the Databricks account and provisions a workspace inside it. +## Prerequisites -## Requirements +- A GCP project with the Databricks platform onboarded +- A service account with workspace-creator role (see `examples/gcp-sa-provisioning`) +- Databricks account ID +- CIDR ranges for the spoke VPC and subnet that don't overlap with existing networks -- You need to have run gcp-sa-provisionning and have a service account to fill in the variables. -- If you want to deploy to a new project, you will need to grant the custom role generated in that template to the service acount in the new project. -- The sizing of the custom vpc subnets needs to be appropriate for the usage of the workspace. [This documentation covers it](https://docs.gcp.databricks.com/administration-guide/cloud-configurations/gcp/network-sizing.html) +## Apply -## Run as an SA +```bash +terraform init +terraform apply +``` -You can do the same thing by provisionning a service account that will have the same permissions - and associate the key associated to it. +## Migrating from the old example +This example previously called `modules/gcp-workspace-byovpc`. Several variable +names changed to match the new composer API: -## Run the tempalte +| Old name | New name | +|----------|----------| +| `subnet_ip_cidr_range` | `subnet_cidr` | +| `pod_ip_cidr_range` | (removed — the GCP data plane runs on GCE; no secondary ranges needed) | +| `svc_ip_cidr_range` | (removed — the GCP data plane runs on GCE; no secondary ranges needed) | +| `subnet_name`, `router_name`, `nat_name` | (removed — composer derives from `prefix` + random suffix) | +| `delegate_from` | (removed — handled by `examples/gcp-sa-provisioning`) | +| _(new)_ | `spoke_vpc_cidr` (VPC primary CIDR, distinct from subnet CIDR) | -- You need to fill in the variables.tf -- run `terraform init` -- run `teraform apply` +State from the old apply does **not** migrate cleanly to the new composer +because resource addresses differ. Re-apply on clean state. ## Requirements -No requirements. +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.5 | +| [databricks](#requirement\_databricks) | ~> 1.81 | +| [google](#requirement\_google) | ~> 6.17 | ## Providers -| Name | Version | -|------|---------| -| [google](#provider\_google) | 4.63.1 | +No providers. ## Modules | Name | Source | Version | |------|--------|---------| -| [gcp-byovpc](#module\_gcp-byovpc) | github.com/databricks/terraform-databricks-examples/modules/gcp-workspace-byovpc | n/a | +| [workspace](#module\_workspace) | ../../modules/gcp/databricks-workspace | n/a | ## Resources -| Name | Type | -|------|------| -| [google_client_config.current](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/client_config) | data source | -| [google_client_openid_userinfo.me](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/client_openid_userinfo) | data source | +No resources. ## Inputs | Name | Description | Type | Default | Required | |------|-------------|------|---------|:--------:| | [databricks\_account\_id](#input\_databricks\_account\_id) | Databricks Account ID | `string` | n/a | yes | -| [databricks\_google\_service\_account](#input\_databricks\_google\_service\_account) | Email of the service account used for deployment | `string` | n/a | yes | -| [delegate\_from](#input\_delegate\_from) | Identities to allow to impersonate created service account (in form of user:user.name@example.com, group:deployers@example.com or serviceAccount:sa1@project.iam.gserviceaccount.com) | `list(string)` | n/a | yes | -| [google\_project](#input\_google\_project) | Google project for VCP/workspace deployment | `string` | n/a | yes | -| [google\_region](#input\_google\_region) | Google region for VCP/workspace deployment | `string` | n/a | yes | -| [google\_zone](#input\_google\_zone) | Zone in GCP region | `string` | n/a | yes | -| [nat\_name](#input\_nat\_name) | Name of the NAT service in compute router | `string` | n/a | yes | -| [pod\_ip\_cidr\_range](#input\_pod\_ip\_cidr\_range) | IP Range for Pods subnet (secondary) | `string` | n/a | yes | -| [prefix](#input\_prefix) | Prefix to use in generated VPC name | `string` | n/a | yes | -| [router\_name](#input\_router\_name) | Name of the compute router to create | `string` | n/a | yes | -| [subnet\_ip\_cidr\_range](#input\_subnet\_ip\_cidr\_range) | IP Range for Nodes subnet (primary) | `string` | n/a | yes | -| [subnet\_name](#input\_subnet\_name) | Name of the subnet to create | `string` | n/a | yes | -| [svc\_ip\_cidr\_range](#input\_svc\_ip\_cidr\_range) | IP Range for Services subnet (secondary) | `string` | n/a | yes | +| [databricks\_google\_service\_account](#input\_databricks\_google\_service\_account) | Service account email used for Databricks provider authentication | `string` | n/a | yes | +| [google\_project](#input\_google\_project) | GCP project where the workspace VPC and resources will be created | `string` | n/a | yes | +| [google\_region](#input\_google\_region) | GCP region for workspace deployment | `string` | n/a | yes | +| [google\_zone](#input\_google\_zone) | GCP zone (used by the google provider) | `string` | n/a | yes | +| [prefix](#input\_prefix) | Prefix used to name generated resources | `string` | n/a | yes | +| [spoke\_vpc\_cidr](#input\_spoke\_vpc\_cidr) | CIDR for the spoke VPC (e.g. 10.0.0.0/16) | `string` | n/a | yes | +| [subnet\_cidr](#input\_subnet\_cidr) | CIDR for the workspace subnet primary range (e.g. 10.0.0.0/22) | `string` | n/a | yes | +| [workspace\_name](#input\_workspace\_name) | Workspace name | `string` | n/a | yes | ## Outputs | Name | Description | |------|-------------| -| [databricks\_host](#output\_databricks\_host) | n/a | -| [databricks\_token](#output\_databricks\_token) | n/a | +| [network\_id](#output\_network\_id) | databricks\_mws\_networks ID | +| [vpc\_id](#output\_vpc\_id) | ID of the spoke VPC created by the module | +| [workspace\_id](#output\_workspace\_id) | Databricks workspace ID | +| [workspace\_url](#output\_workspace\_url) | Databricks workspace URL | diff --git a/examples/gcp-byovpc/main.tf b/examples/gcp-byovpc/main.tf index c1e82a06..009e6e6a 100644 --- a/examples/gcp-byovpc/main.tf +++ b/examples/gcp-byovpc/main.tf @@ -1,15 +1,13 @@ -module "gcp-byovpc" { - source = "github.com/databricks/terraform-databricks-examples/modules/gcp-workspace-byovpc" +module "workspace" { + source = "../../modules/gcp/databricks-workspace" + + prefix = var.prefix databricks_account_id = var.databricks_account_id google_project = var.google_project google_region = var.google_region - prefix = var.prefix - subnet_ip_cidr_range = var.subnet_ip_cidr_range - pod_ip_cidr_range = var.pod_ip_cidr_range - svc_ip_cidr_range = var.svc_ip_cidr_range - subnet_name = var.subnet_name - router_name = var.router_name - nat_name = var.nat_name workspace_name = var.workspace_name - delegate_from = var.delegate_from + + vpc_source = { spoke = "create" } + spoke_vpc_cidr = var.spoke_vpc_cidr + subnet_cidr = var.subnet_cidr } diff --git a/examples/gcp-byovpc/outputs.tf b/examples/gcp-byovpc/outputs.tf index f544b3ba..202343f0 100644 --- a/examples/gcp-byovpc/outputs.tf +++ b/examples/gcp-byovpc/outputs.tf @@ -1,8 +1,19 @@ -output "databricks_host" { - value = databricks_mws_workspaces.databricks_workspace.workspace_url +output "workspace_id" { + value = module.workspace.workspace_id + description = "Databricks workspace ID" } -output "databricks_token" { - value = databricks_mws_workspaces.databricks_workspace.token[0].token_value - sensitive = true +output "workspace_url" { + value = module.workspace.workspace_url + description = "Databricks workspace URL" +} + +output "vpc_id" { + value = module.workspace.spoke_vpc_id + description = "ID of the spoke VPC created by the module" +} + +output "network_id" { + value = module.workspace.network_id + description = "databricks_mws_networks ID" } \ No newline at end of file diff --git a/examples/gcp-byovpc/init.tf b/examples/gcp-byovpc/providers.tf similarity index 55% rename from examples/gcp-byovpc/init.tf rename to examples/gcp-byovpc/providers.tf index 55e8f98c..edead550 100644 --- a/examples/gcp-byovpc/init.tf +++ b/examples/gcp-byovpc/providers.tf @@ -1,31 +1,11 @@ -terraform { - required_providers { - databricks = { - source = "databricks/databricks" - } - google = { - source = "hashicorp/google" - } - } -} - provider "google" { project = var.google_project region = var.google_region zone = var.google_zone - } provider "databricks" { host = "https://accounts.gcp.databricks.com" google_service_account = var.databricks_google_service_account account_id = var.databricks_account_id - } - -data "google_client_openid_userinfo" "me" { -} - - -data "google_client_config" "current" { -} \ No newline at end of file diff --git a/examples/gcp-byovpc/terraform.tfvars b/examples/gcp-byovpc/terraform.tfvars new file mode 100644 index 00000000..5c47e9b7 --- /dev/null +++ b/examples/gcp-byovpc/terraform.tfvars @@ -0,0 +1,9 @@ +databricks_account_id = "" +databricks_google_service_account = "" +google_project = "" +google_region = "" +google_zone = "" +prefix = "" +workspace_name = "" +spoke_vpc_cidr = "" +subnet_cidr = "" diff --git a/examples/gcp-byovpc/variables.tf b/examples/gcp-byovpc/variables.tf index e1c91f2d..b5bc8d71 100644 --- a/examples/gcp-byovpc/variables.tf +++ b/examples/gcp-byovpc/variables.tf @@ -4,61 +4,41 @@ variable "databricks_account_id" { } variable "databricks_google_service_account" { - description = "Email of the service account used for deployment" type = string + description = "Service account email used for Databricks provider authentication" } variable "google_project" { type = string - description = "Google project for VCP/workspace deployment" + description = "GCP project where the workspace VPC and resources will be created" } variable "google_region" { type = string - description = "Google region for VCP/workspace deployment" + description = "GCP region for workspace deployment" } variable "google_zone" { - description = "Zone in GCP region" type = string + description = "GCP zone (used by the google provider)" } variable "prefix" { type = string - description = "Prefix to use in generated VPC name" + description = "Prefix used to name generated resources" } -variable "subnet_ip_cidr_range" { +variable "workspace_name" { type = string - description = "IP Range for Nodes subnet (primary)" + description = "Workspace name" } -variable "pod_ip_cidr_range" { +variable "spoke_vpc_cidr" { type = string - description = "IP Range for Pods subnet (secondary)" + description = "CIDR for the spoke VPC (e.g. 10.0.0.0/16)" } -variable "svc_ip_cidr_range" { +variable "subnet_cidr" { type = string - description = "IP Range for Services subnet (secondary)" -} - -variable "subnet_name" { - type = string - description = "Name of the subnet to create" -} - -variable "router_name" { - type = string - description = "Name of the compute router to create" -} - -variable "nat_name" { - type = string - description = "Name of the NAT service in compute router" -} - -variable "delegate_from" { - description = "Identities to allow to impersonate created service account (in form of user:user.name@example.com, group:deployers@example.com or serviceAccount:sa1@project.iam.gserviceaccount.com)" - type = list(string) + description = "CIDR for the workspace subnet primary range (e.g. 10.0.0.0/22)" } diff --git a/examples/gcp-byovpc/versions.tf b/examples/gcp-byovpc/versions.tf new file mode 100644 index 00000000..150d0ab5 --- /dev/null +++ b/examples/gcp-byovpc/versions.tf @@ -0,0 +1,13 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { + source = "databricks/databricks" + version = "~> 1.81" + } + google = { + source = "hashicorp/google" + version = "~> 6.17" + } + } +} diff --git a/examples/gcp-sa-provisionning/Makefile b/examples/gcp-existing-vpc/Makefile similarity index 100% rename from examples/gcp-sa-provisionning/Makefile rename to examples/gcp-existing-vpc/Makefile diff --git a/examples/gcp-existing-vpc/README.md b/examples/gcp-existing-vpc/README.md new file mode 100644 index 00000000..b47396cd --- /dev/null +++ b/examples/gcp-existing-vpc/README.md @@ -0,0 +1,78 @@ +# examples/gcp-existing-vpc — Use a pre-existing VPC + +Calls `modules/gcp/databricks-workspace` with `vpc_source = { spoke = "existing" }`. Instead +of creating a VPC, the composer looks up the named VPC + subnet via Terraform +data sources and registers them with the Databricks account. + +This is the scenario for organizations that manage GCP networking out-of-band +(e.g. via a platform team) and just want Databricks to consume an existing +network. + +## Prerequisites + +- A GCP project with the Databricks platform onboarded +- A pre-existing VPC and subnet in that project. The subnet must be in `google_region`. +- A service account with workspace-creator role (see `examples/gcp-sa-provisioning`) +- Databricks account ID + +## Apply + +```bash +terraform init +terraform apply +``` + +## What the composer does NOT do in this mode + +- Does not create the VPC, subnet, router, or NAT — those must already exist +- Does not enforce that the subnet has Private Google Access enabled — verify in the console +- Does not configure egress firewalls or PrivateLink (those require `vpc_source = { spoke = "create" }`) + +To layer PrivateLink onto an existing network, the current composer requires +`vpc_source = { spoke = "create" }`. Future work may relax this. + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.5 | +| [databricks](#requirement\_databricks) | ~> 1.81 | +| [google](#requirement\_google) | ~> 6.17 | + +## Providers + +No providers. + +## Modules + +| Name | Source | Version | +|------|--------|---------| +| [workspace](#module\_workspace) | ../../modules/gcp/databricks-workspace | n/a | + +## Resources + +No resources. + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [databricks\_account\_id](#input\_databricks\_account\_id) | Databricks Account ID | `string` | n/a | yes | +| [databricks\_google\_service\_account](#input\_databricks\_google\_service\_account) | Service account email used for Databricks provider authentication | `string` | n/a | yes | +| [existing\_subnet\_name](#input\_existing\_subnet\_name) | Name of the pre-existing subnet inside the VPC (must be in google\_region) | `string` | n/a | yes | +| [existing\_vpc\_name](#input\_existing\_vpc\_name) | Name of the pre-existing GCP VPC to deploy the workspace into | `string` | n/a | yes | +| [google\_project](#input\_google\_project) | GCP project hosting the existing VPC and subnet (also the workspace project) | `string` | n/a | yes | +| [google\_region](#input\_google\_region) | GCP region for workspace deployment (must match the existing subnet's region) | `string` | n/a | yes | +| [google\_zone](#input\_google\_zone) | GCP zone (used by the google provider) | `string` | n/a | yes | +| [prefix](#input\_prefix) | Prefix used to name Databricks-side resources (mws\_networks, mws\_workspaces) | `string` | n/a | yes | +| [workspace\_name](#input\_workspace\_name) | Workspace name | `string` | n/a | yes | + +## Outputs + +| Name | Description | +|------|-------------| +| [network\_id](#output\_network\_id) | databricks\_mws\_networks ID | +| [workspace\_id](#output\_workspace\_id) | Databricks workspace ID | +| [workspace\_url](#output\_workspace\_url) | Databricks workspace URL | + diff --git a/examples/gcp-existing-vpc/main.tf b/examples/gcp-existing-vpc/main.tf new file mode 100644 index 00000000..7e072046 --- /dev/null +++ b/examples/gcp-existing-vpc/main.tf @@ -0,0 +1,13 @@ +module "workspace" { + source = "../../modules/gcp/databricks-workspace" + + prefix = var.prefix + databricks_account_id = var.databricks_account_id + google_project = var.google_project + google_region = var.google_region + workspace_name = var.workspace_name + + vpc_source = { spoke = "existing" } + existing_vpc_name = var.existing_vpc_name + existing_subnet_name = var.existing_subnet_name +} diff --git a/examples/gcp-existing-vpc/outputs.tf b/examples/gcp-existing-vpc/outputs.tf new file mode 100644 index 00000000..469a66e6 --- /dev/null +++ b/examples/gcp-existing-vpc/outputs.tf @@ -0,0 +1,14 @@ +output "workspace_id" { + value = module.workspace.workspace_id + description = "Databricks workspace ID" +} + +output "workspace_url" { + value = module.workspace.workspace_url + description = "Databricks workspace URL" +} + +output "network_id" { + value = module.workspace.network_id + description = "databricks_mws_networks ID" +} diff --git a/examples/gcp-existing-vpc/providers.tf b/examples/gcp-existing-vpc/providers.tf new file mode 100644 index 00000000..edead550 --- /dev/null +++ b/examples/gcp-existing-vpc/providers.tf @@ -0,0 +1,11 @@ +provider "google" { + project = var.google_project + region = var.google_region + zone = var.google_zone +} + +provider "databricks" { + host = "https://accounts.gcp.databricks.com" + google_service_account = var.databricks_google_service_account + account_id = var.databricks_account_id +} diff --git a/examples/gcp-existing-vpc/terraform.tfvars b/examples/gcp-existing-vpc/terraform.tfvars new file mode 100644 index 00000000..a541640e --- /dev/null +++ b/examples/gcp-existing-vpc/terraform.tfvars @@ -0,0 +1,9 @@ +databricks_account_id = "" +databricks_google_service_account = "" +google_project = "" +google_region = "" +google_zone = "" +prefix = "" +workspace_name = "" +existing_vpc_name = "" +existing_subnet_name = "" diff --git a/examples/gcp-existing-vpc/variables.tf b/examples/gcp-existing-vpc/variables.tf new file mode 100644 index 00000000..f0518c88 --- /dev/null +++ b/examples/gcp-existing-vpc/variables.tf @@ -0,0 +1,44 @@ +variable "databricks_account_id" { + type = string + description = "Databricks Account ID" +} + +variable "databricks_google_service_account" { + type = string + description = "Service account email used for Databricks provider authentication" +} + +variable "google_project" { + type = string + description = "GCP project hosting the existing VPC and subnet (also the workspace project)" +} + +variable "google_region" { + type = string + description = "GCP region for workspace deployment (must match the existing subnet's region)" +} + +variable "google_zone" { + type = string + description = "GCP zone (used by the google provider)" +} + +variable "prefix" { + type = string + description = "Prefix used to name Databricks-side resources (mws_networks, mws_workspaces)" +} + +variable "workspace_name" { + type = string + description = "Workspace name" +} + +variable "existing_vpc_name" { + type = string + description = "Name of the pre-existing GCP VPC to deploy the workspace into" +} + +variable "existing_subnet_name" { + type = string + description = "Name of the pre-existing subnet inside the VPC (must be in google_region)" +} diff --git a/examples/gcp-existing-vpc/versions.tf b/examples/gcp-existing-vpc/versions.tf new file mode 100644 index 00000000..150d0ab5 --- /dev/null +++ b/examples/gcp-existing-vpc/versions.tf @@ -0,0 +1,13 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { + source = "databricks/databricks" + version = "~> 1.81" + } + google = { + source = "hashicorp/google" + version = "~> 6.17" + } + } +} diff --git a/examples/gcp-sa-provisioning/README.md b/examples/gcp-sa-provisioning/README.md index 48304ca1..eda4b54a 100644 --- a/examples/gcp-sa-provisioning/README.md +++ b/examples/gcp-sa-provisioning/README.md @@ -32,28 +32,24 @@ No requirements. ## Providers -| Name | Version | -|------|---------| -| [google](#provider\_google) | n/a | +No providers. ## Modules | Name | Source | Version | |------|--------|---------| -| [gcp-sa-provisioning](#module\_gcp-sa-provisioning) | github.com/databricks/terraform-databricks-examples/modules/gcp-sa-provisioning | n/a | +| [service\_account](#module\_service\_account) | ../../modules/gcp/service-account | n/a | ## Resources -| Name | Type | -|------|------| -| [google_client_openid_userinfo.me](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/client_openid_userinfo) | data source | +No resources. ## Inputs | Name | Description | Type | Default | Required | |------|-------------|------|---------|:--------:| -| [delegate\_from](#input\_delegate\_from) | Identities to allow to impersonate created service account (in form of user:user.name@example.com, group:deployers@example.com or serviceAccount:sa1@project.iam.gserviceaccount.com) | `list(string)` | n/a | yes | -| [google\_project](#input\_google\_project) | Google project for VCP/workspace deployment | `string` | n/a | yes | +| [delegate\_from](#input\_delegate\_from) | Identities to allow to impersonate created service account (in form of user:user.name@example.com, group:deployers@example.com or serviceAccount:deployer@my-project.iam.gserviceaccount.com) | `list(string)` | n/a | yes | +| [google\_project](#input\_google\_project) | Google project for VPC/workspace deployment | `string` | n/a | yes | | [google\_region](#input\_google\_region) | GCP region for deployment | `string` | n/a | yes | | [google\_zone](#input\_google\_zone) | Zone in GCP region | `string` | n/a | yes | | [prefix](#input\_prefix) | Prefix to use in generated service account name. This should not contain underscores or dashes. | `string` | n/a | yes | diff --git a/examples/gcp-sa-provisioning/init.tf b/examples/gcp-sa-provisioning/init.tf deleted file mode 100644 index 5332dab6..00000000 --- a/examples/gcp-sa-provisioning/init.tf +++ /dev/null @@ -1,17 +0,0 @@ -terraform { - required_providers { - - google = { - source = "hashicorp/google" - } - } -} - -provider "google" { - project = var.google_project - region = var.google_region - zone = var.google_zone - -} -data "google_client_openid_userinfo" "me" { -} diff --git a/examples/gcp-sa-provisioning/main.tf b/examples/gcp-sa-provisioning/main.tf index 7b596530..0e6f2d4b 100644 --- a/examples/gcp-sa-provisioning/main.tf +++ b/examples/gcp-sa-provisioning/main.tf @@ -1,5 +1,5 @@ -module "gcp-sa-provisioning" { - source = "github.com/databricks/terraform-databricks-examples/modules/gcp-sa-provisioning" +module "service_account" { + source = "../../modules/gcp/service-account" google_project = var.google_project prefix = var.prefix delegate_from = var.delegate_from diff --git a/examples/gcp-sa-provisioning/outputs.tf b/examples/gcp-sa-provisioning/outputs.tf index 5fec4b42..3f7d5d8e 100644 --- a/examples/gcp-sa-provisioning/outputs.tf +++ b/examples/gcp-sa-provisioning/outputs.tf @@ -1,4 +1,4 @@ output "service_account" { - value = module.gcp-sa-provisioning.service_account + value = module.service_account.service_account_email description = "Add this email as a user in the Databricks account console" } diff --git a/examples/gcp-sa-provisioning/providers.tf b/examples/gcp-sa-provisioning/providers.tf new file mode 100644 index 00000000..6040c599 --- /dev/null +++ b/examples/gcp-sa-provisioning/providers.tf @@ -0,0 +1,5 @@ +provider "google" { + project = var.google_project + region = var.google_region + zone = var.google_zone +} diff --git a/examples/gcp-sa-provisioning/variables.tf b/examples/gcp-sa-provisioning/variables.tf index c4dfebde..bbccb3c7 100644 --- a/examples/gcp-sa-provisioning/variables.tf +++ b/examples/gcp-sa-provisioning/variables.tf @@ -1,6 +1,6 @@ variable "google_project" { type = string - description = "Google project for VCP/workspace deployment" + description = "Google project for VPC/workspace deployment" } variable "prefix" { @@ -9,7 +9,7 @@ variable "prefix" { } variable "delegate_from" { - description = "Identities to allow to impersonate created service account (in form of user:user.name@example.com, group:deployers@example.com or serviceAccount:sa1@project.iam.gserviceaccount.com)" + description = "Identities to allow to impersonate created service account (in form of user:user.name@example.com, group:deployers@example.com or serviceAccount:deployer@my-project.iam.gserviceaccount.com)" type = list(string) } diff --git a/modules/gcp-sa-provisioning/init.tf b/examples/gcp-sa-provisioning/versions.tf similarity index 66% rename from modules/gcp-sa-provisioning/init.tf rename to examples/gcp-sa-provisioning/versions.tf index 3cd60564..b3340e11 100644 --- a/modules/gcp-sa-provisioning/init.tf +++ b/examples/gcp-sa-provisioning/versions.tf @@ -1,11 +1,7 @@ terraform { required_providers { - google = { source = "hashicorp/google" } } } - -data "google_client_openid_userinfo" "me" { -} diff --git a/examples/gcp-with-psc-exfiltration-protection/README.md b/examples/gcp-with-psc-exfiltration-protection/README.md index 64d676a9..85f1fbb5 100644 --- a/examples/gcp-with-psc-exfiltration-protection/README.md +++ b/examples/gcp-with-psc-exfiltration-protection/README.md @@ -1,45 +1,59 @@ -# Provisioning Databricks on GCP workspace with a Hub & Spoke network architecture for data exfiltration protection +# examples/gcp-with-psc-exfiltration-protection — Workspace with PSC + private DNS + restricted egress -This example is using the [gcp-with-psc-exfiltration-protection](../../modules/gcp-with-psc-exfiltration-protection) module. +Calls `modules/gcp/databricks-workspace` with all PrivateLink and egress-control flags enabled: -This template provides an example deployment of: Hub-Spoke networking with egress firewall to control all outbound traffic from Databricks subnets. +- `vpc_source = { spoke = "create", hub = "create" }` — composer creates the spoke VPC + hub VPC + peering +- `private_link_frontend = true` — frontend PSC endpoint (workspace UI/API) +- `private_link_backend = true` — backend (SCC) PSC endpoint (data plane) +- `private_access_only = true` — `mws_private_access_settings.public_access_enabled = false` +- `restricted_egress = true` — hub VPC + deny-egress firewall + private DNS zones -With this setup, you can setup firewall rules to block / allow egress traffic from your Databricks clusters. You can also use firewall to block all access to storage accounts, and use private endpoint connection to bypass this firewall, such that you allow access only to specific storage accounts. +Optionally pairs with the `modules/gcp/unity-catalog` module to create a metastore, GCS bucket, storage credential, external location, and default catalog. +By default this example sets `serverless_egress_mode = "restricted"`, so serverless workloads are deny-by-default and must be explicitly allow-listed via `serverless_allowed_internet_destinations`/`serverless_allowed_storage_destinations` (matching the classic-compute egress posture set up by `restricted_egress`); set it to `"unmanaged"` to opt out. The example also exposes optional CMEK (`cmek_managed_services_key_id`, `cmek_storage_key_id`) and security-settings variables (Compliance Security Profile, Enhanced Security Monitoring, IP access lists) — all null/disabled unless set. -To find IP and FQDN for your deployment, go to: https://docs.gcp.databricks.com/en/resources/ip-domain-region.html +## Prerequisites -## Overall Architecture +- Two (or three) GCP projects: workspace project, spoke VPC project, hub VPC project (can be the same) +- Service account with workspace-creator role (see `examples/gcp-sa-provisioning`) +- Databricks account ID +- CIDR ranges that don't overlap: `spoke_vpc_cidr`, `subnet_cidr` (subset of spoke), `hub_vpc_cidr`, `psc_subnet_cidr` -![alt text](../../modules/gcp-with-psc-exfiltration-protection/images/architecture.png) +## Apply -Resources to be created: -* Hub VPC and its subnet -* Spoke VPC and its subnets -* Peering between Hub and Spoke VPC -* Private Service Connect (PSC) endpoints -* DNS private and peering zones -* Firewall rules for Hub and Spoke VPCs -* Databricks workspace with private link to control plane, user to webapp and private link to DBFS +```bash +terraform init +terraform apply +``` -## How to use +## Migrating from the old example -1. Reference this module using one of the different [module source types](https://developer.hashicorp.com/terraform/language/modules/sources) -2. Add `terraform.tfvars` with the information about service principals to be provisioned at account level. +This example previously called `modules/gcp-with-psc-exfiltration-protection` and `modules/gcp-unity-catalog`. Key changes: -## How to fill in variable values +| Old | New | +|-----|-----| +| `module.gcp_with_data_exfiltration_protection` | `module.workspace` | +| `modules/gcp-with-psc-exfiltration-protection` | `modules/gcp/databricks-workspace` with `vpc_source={spoke="create",hub="create"}` + 4 PSC/egress flags | +| `modules/gcp-unity-catalog` | `modules/gcp/unity-catalog` (relocated, same interface) | +| `spoke_vpc_cidr` (legacy: was used as subnet CIDR AND firewall source ranges) | Split into `subnet_cidr` (subnet CIDR) and `spoke_vpc_cidr` (broader VPC CIDR for firewall source) | -Variables have no default values in order to avoid misconfiguration +State from the old apply does **not** migrate cleanly to the new composer because resource addresses differ. Re-apply on clean state. -Most values are related to resources managed by Databricks. The required values can be found at: https://docs.gcp.databricks.com/en/resources/ip-domain-region.html +## Running this example + +- Both the `databricks` (account-level) and `databricks` (workspace-level, used by `unity_catalog`/`security_settings`) provider configurations authenticate via Google Application Default Credentials — run `gcloud auth application-default login`, or use `google_service_account` impersonation. The module no longer creates or accepts a Databricks PAT. +- With `private_access_only = true` (the default here), the workspace has no public endpoint: the Unity Catalog and security-settings steps talk to the workspace over its URL, so the machine running `terraform apply` needs network reachability to the frontend PSC endpoint. Run it from a VM inside the hub/spoke network (or anything connected to it via VPN/peering) — not from an arbitrary laptop on the public internet. +- Serverless egress control (`serverless_egress_mode`) and CMEK (`cmek_managed_services_key_id`/`cmek_storage_key_id`) both require the workspace to be on the Enterprise tier. ## Requirements | Name | Version | |------|---------| -| [databricks](#requirement\_databricks) | >=1.81.1 | -| [google](#requirement\_google) | 6.17.0 | +| [terraform](#requirement\_terraform) | >= 1.5 | +| [databricks](#requirement\_databricks) | ~> 1.81 | +| [google](#requirement\_google) | ~> 6.17 | +| [random](#requirement\_random) | >= 3.0 | ## Providers @@ -49,8 +63,9 @@ No providers. | Name | Source | Version | |------|--------|---------| -| [gcp\_with\_data\_exfiltration\_protection](#module\_gcp\_with\_data\_exfiltration\_protection) | ../../modules/gcp-with-psc-exfiltration-protection | n/a | -| [unity\_catalog](#module\_unity\_catalog) | ../../modules/gcp-unity-catalog | n/a | +| [security\_settings](#module\_security\_settings) | ../../modules/databricks/security-settings | n/a | +| [unity\_catalog](#module\_unity\_catalog) | ../../modules/gcp/unity-catalog | n/a | +| [workspace](#module\_workspace) | ../../modules/gcp/databricks-workspace | n/a | ## Resources @@ -60,25 +75,39 @@ No resources. | Name | Description | Type | Default | Required | |------|-------------|------|---------|:--------:| -| [catalog\_name](#input\_catalog\_name) | Name to assign to default catalog | `string` | n/a | yes | +| [catalog\_name](#input\_catalog\_name) | Name to assign to default Unity Catalog catalog | `string` | n/a | yes | | [databricks\_account\_id](#input\_databricks\_account\_id) | Databricks Account ID | `string` | n/a | yes | | [google\_region](#input\_google\_region) | Google Cloud region where the resources will be created | `string` | n/a | yes | -| [hive\_metastore\_ip](#input\_hive\_metastore\_ip) | Value of regional default Hive Metastore IP | `string` | n/a | yes | -| [hub\_vpc\_cidr](#input\_hub\_vpc\_cidr) | CIDR for Hub VPC | `string` | n/a | yes | -| [hub\_vpc\_google\_project](#input\_hub\_vpc\_google\_project) | Google Cloud project ID related to Hub VPC | `string` | n/a | yes | -| [is\_spoke\_vpc\_shared](#input\_is\_spoke\_vpc\_shared) | Whether the Spoke VPC is a Shared or a dedicated VPC | `bool` | n/a | yes | -| [metastore\_name](#input\_metastore\_name) | Name to assign to regional metastore | `string` | n/a | yes | -| [prefix](#input\_prefix) | Prefix to use in generated resources name | `string` | n/a | yes | -| [psc\_subnet\_cidr](#input\_psc\_subnet\_cidr) | CIDR for Spoke VPC | `string` | n/a | yes | -| [spoke\_vpc\_cidr](#input\_spoke\_vpc\_cidr) | CIDR for Spoke VPC | `string` | n/a | yes | -| [spoke\_vpc\_google\_project](#input\_spoke\_vpc\_google\_project) | Google Cloud project ID related to Spoke VPC | `string` | n/a | yes | -| [workspace\_google\_project](#input\_workspace\_google\_project) | Google Cloud project ID related to Databricks workspace | `string` | n/a | yes | -| [tags](#input\_tags) | Map of tags to add to all resources | `map(string)` | `{}` | no | +| [hub\_vpc\_cidr](#input\_hub\_vpc\_cidr) | CIDR for the hub subnet | `string` | n/a | yes | +| [hub\_vpc\_google\_project](#input\_hub\_vpc\_google\_project) | Google Cloud project ID hosting the hub VPC | `string` | n/a | yes | +| [is\_spoke\_vpc\_shared](#input\_is\_spoke\_vpc\_shared) | Whether the spoke VPC project hosts a Shared VPC and the workspace project is bound as a service project | `bool` | n/a | yes | +| [metastore\_name](#input\_metastore\_name) | Name to assign to regional Unity Catalog metastore | `string` | n/a | yes | +| [prefix](#input\_prefix) | Prefix used to name generated resources | `string` | n/a | yes | +| [psc\_subnet\_cidr](#input\_psc\_subnet\_cidr) | CIDR for the dedicated PSC subnet in the spoke VPC | `string` | n/a | yes | +| [spoke\_vpc\_cidr](#input\_spoke\_vpc\_cidr) | CIDR of the spoke VPC address space (used as source\_ranges for the hub ingress firewall) | `string` | n/a | yes | +| [spoke\_vpc\_google\_project](#input\_spoke\_vpc\_google\_project) | Google Cloud project ID hosting the spoke VPC (often the same as workspace project) | `string` | n/a | yes | +| [subnet\_cidr](#input\_subnet\_cidr) | CIDR for the spoke subnet (must be within spoke\_vpc\_cidr) | `string` | n/a | yes | +| [workspace\_google\_project](#input\_workspace\_google\_project) | Google Cloud project ID where the Databricks workspace lives | `string` | n/a | yes | +| [cmek\_managed\_services\_key\_id](#input\_cmek\_managed\_services\_key\_id) | Cloud KMS key resource ID for managed-services CMEK (control-plane data: notebooks, secrets, queries). Null disables. The principal running Terraform needs cloudkms.cryptoKeys.getIamPolicy and setIamPolicy on the key - Databricks sets the key's IAM policy at workspace creation. Enterprise tier; set at creation only. The key must exist before plan (a key created in the same configuration makes the count unknown and fails plan) | `string` | `null` | no | +| [cmek\_storage\_key\_id](#input\_cmek\_storage\_key\_id) | Cloud KMS key resource ID for workspace-storage CMEK (GCS buckets and GCE persistent disks). Null disables. Same permission and tier requirements as cmek\_managed\_services\_key\_id; set at creation only. The key must exist before plan (a key created in the same configuration makes the count unknown and fails plan) | `string` | `null` | no | +| [compliance\_standards](#input\_compliance\_standards) | Compliance standards for the CSP (e.g. ["HIPAA"]). Only meaningful when enable\_compliance\_security\_profile=true | `list(string)` | `[]` | no | +| [enable\_automatic\_cluster\_update](#input\_enable\_automatic\_cluster\_update) | Enable automatic cluster update for the workspace | `bool` | `false` | no | +| [enable\_compliance\_security\_profile](#input\_enable\_compliance\_security\_profile) | Enable the Compliance Security Profile on the workspace. WARNING: irreversible - CSP cannot be disabled once enabled. Requires enable\_enhanced\_security\_monitoring=true | `bool` | `false` | no | +| [enable\_enhanced\_security\_monitoring](#input\_enable\_enhanced\_security\_monitoring) | Enable Enhanced Security Monitoring (hardened images, monitoring agents) | `bool` | `false` | no | +| [hive\_metastore\_ip](#input\_hive\_metastore\_ip) | Regional legacy Hive metastore IP. When set, an egress allow rule (tcp/3306) is created under restricted egress; when null, no rule is created. Workspaces using Unity Catalog (the default) do not need this. Regional IPs: https://docs.databricks.com/gcp/en/resources/ip-domain-region | `string` | `null` | no | +| [ip\_access\_lists](#input\_ip\_access\_lists) | Workspace IP access lists. list\_type is ALLOW or BLOCK. A non-empty list also flips the enableIpAccessLists workspace conf | 
list(object({
    label        = string
    list_type    = string
    ip_addresses = list(string)
  }))
| `[]` | no | +| [serverless\_allowed\_internet\_destinations](#input\_serverless\_allowed\_internet\_destinations) | FQDNs serverless workloads may reach (only with serverless\_egress\_mode=restricted) | `list(string)` | `[]` | no | +| [serverless\_allowed\_storage\_destinations](#input\_serverless\_allowed\_storage\_destinations) | GCS bucket names serverless workloads may reach (only with serverless\_egress\_mode=restricted) | `list(string)` | `[]` | no | +| [serverless\_egress\_enforcement](#input\_serverless\_egress\_enforcement) | enforced or dry\_run (log-only evaluation) | `string` | `"enforced"` | no | +| [serverless\_egress\_mode](#input\_serverless\_egress\_mode) | Serverless egress control mode (unmanaged, full, restricted). Default restricted: deny-by-default for serverless, matching this example's classic-compute posture. Requires Enterprise tier | `string` | `"restricted"` | no | ## Outputs | Name | Description | |------|-------------| +| [hub\_vpc\_id](#output\_hub\_vpc\_id) | ID of the hub VPC | +| [network\_id](#output\_network\_id) | databricks\_mws\_networks ID | +| [vpc\_id](#output\_vpc\_id) | ID of the spoke VPC | | [workspace\_id](#output\_workspace\_id) | The Databricks workspace ID | | [workspace\_url](#output\_workspace\_url) | The workspace URL which is of the format '{workspaceId}.{random}.gcp.databricks.com' | diff --git a/examples/gcp-with-psc-exfiltration-protection/main.tf b/examples/gcp-with-psc-exfiltration-protection/main.tf index c0b7fa91..34dc12f5 100644 --- a/examples/gcp-with-psc-exfiltration-protection/main.tf +++ b/examples/gcp-with-psc-exfiltration-protection/main.tf @@ -1,16 +1,32 @@ -module "gcp_with_data_exfiltration_protection" { - source = "../../modules/gcp-with-psc-exfiltration-protection" +module "workspace" { + source = "../../modules/gcp/databricks-workspace" - databricks_account_id = var.databricks_account_id + prefix = var.prefix + databricks_account_id = var.databricks_account_id + google_project = var.workspace_google_project + google_region = var.google_region + + vpc_source = { spoke = "create", hub = "create" } + spoke_vpc_cidr = var.spoke_vpc_cidr + subnet_cidr = var.subnet_cidr + + private_link_frontend = true + private_link_backend = true + private_access_only = true + restricted_egress = true + + spoke_vpc_google_project = var.spoke_vpc_google_project hub_vpc_google_project = var.hub_vpc_google_project is_spoke_vpc_shared = var.is_spoke_vpc_shared - prefix = var.prefix - spoke_vpc_google_project = var.spoke_vpc_google_project - workspace_google_project = var.workspace_google_project - google_region = var.google_region - hive_metastore_ip = var.hive_metastore_ip hub_vpc_cidr = var.hub_vpc_cidr psc_subnet_cidr = var.psc_subnet_cidr - spoke_vpc_cidr = var.spoke_vpc_cidr - tags = var.tags + hive_metastore_ip = var.hive_metastore_ip + + serverless_egress_mode = var.serverless_egress_mode + serverless_allowed_internet_destinations = var.serverless_allowed_internet_destinations + serverless_allowed_storage_destinations = var.serverless_allowed_storage_destinations + serverless_egress_enforcement = var.serverless_egress_enforcement + + cmek_managed_services_key_id = var.cmek_managed_services_key_id + cmek_storage_key_id = var.cmek_storage_key_id } \ No newline at end of file diff --git a/examples/gcp-with-psc-exfiltration-protection/outputs.tf b/examples/gcp-with-psc-exfiltration-protection/outputs.tf index 681fe5d0..76e7c517 100644 --- a/examples/gcp-with-psc-exfiltration-protection/outputs.tf +++ b/examples/gcp-with-psc-exfiltration-protection/outputs.tf @@ -1,10 +1,24 @@ - output "workspace_url" { - value = module.gcp_with_data_exfiltration_protection.workspace_url + value = module.workspace.workspace_url description = "The workspace URL which is of the format '{workspaceId}.{random}.gcp.databricks.com'" } output "workspace_id" { + value = module.workspace.workspace_id description = "The Databricks workspace ID" - value = module.gcp_with_data_exfiltration_protection.workspace_id +} + +output "vpc_id" { + value = module.workspace.spoke_vpc_id + description = "ID of the spoke VPC" +} + +output "hub_vpc_id" { + value = module.workspace.hub_vpc_id + description = "ID of the hub VPC" +} + +output "network_id" { + value = module.workspace.network_id + description = "databricks_mws_networks ID" } \ No newline at end of file diff --git a/examples/gcp-with-psc-exfiltration-protection/providers.tf b/examples/gcp-with-psc-exfiltration-protection/providers.tf index 489bf1e9..f2881ffd 100644 --- a/examples/gcp-with-psc-exfiltration-protection/providers.tf +++ b/examples/gcp-with-psc-exfiltration-protection/providers.tf @@ -6,7 +6,7 @@ provider "databricks" { provider "databricks" { alias = "workspace" - host = module.gcp_with_data_exfiltration_protection.workspace_url + host = module.workspace.workspace_url } provider "google" { diff --git a/examples/gcp-with-psc-exfiltration-protection/security-settings.tf b/examples/gcp-with-psc-exfiltration-protection/security-settings.tf new file mode 100644 index 00000000..9253d755 --- /dev/null +++ b/examples/gcp-with-psc-exfiltration-protection/security-settings.tf @@ -0,0 +1,15 @@ +module "security_settings" { + source = "../../modules/databricks/security-settings" + + providers = { + databricks = databricks.workspace + } + + enable_compliance_security_profile = var.enable_compliance_security_profile + compliance_standards = var.compliance_standards + enable_enhanced_security_monitoring = var.enable_enhanced_security_monitoring + enable_automatic_cluster_update = var.enable_automatic_cluster_update + ip_access_lists = var.ip_access_lists + + depends_on = [module.workspace] +} diff --git a/examples/gcp-with-psc-exfiltration-protection/terraform.tfvars b/examples/gcp-with-psc-exfiltration-protection/terraform.tfvars index 8f095727..7b96dbf1 100644 --- a/examples/gcp-with-psc-exfiltration-protection/terraform.tfvars +++ b/examples/gcp-with-psc-exfiltration-protection/terraform.tfvars @@ -10,11 +10,24 @@ is_spoke_vpc_shared = true prefix = "" -hive_metastore_ip = "" -hub_vpc_cidr = "" -spoke_vpc_cidr = "" -psc_subnet_cidr = "" +hub_vpc_cidr = "" +spoke_vpc_cidr = "" +subnet_cidr = "" +psc_subnet_cidr = "" metastore_name = "" catalog_name = "" +serverless_egress_mode = "restricted" +serverless_allowed_internet_destinations = [] +serverless_allowed_storage_destinations = [] +serverless_egress_enforcement = "enforced" + +cmek_managed_services_key_id = null +cmek_storage_key_id = null + +enable_compliance_security_profile = false +compliance_standards = [] +enable_enhanced_security_monitoring = false +enable_automatic_cluster_update = false +ip_access_lists = [] diff --git a/examples/gcp-with-psc-exfiltration-protection/unity-catalog.tf b/examples/gcp-with-psc-exfiltration-protection/unity-catalog.tf index c6c0628c..792862d8 100644 --- a/examples/gcp-with-psc-exfiltration-protection/unity-catalog.tf +++ b/examples/gcp-with-psc-exfiltration-protection/unity-catalog.tf @@ -1,15 +1,16 @@ module "unity_catalog" { - source = "../../modules/gcp-unity-catalog" + source = "../../modules/gcp/unity-catalog" providers = { - databricks = databricks, + databricks = databricks databricks.workspace = databricks.workspace } - databricks_workspace_id = module.gcp_with_data_exfiltration_protection.workspace_id - databricks_workspace_url = module.gcp_with_data_exfiltration_protection.workspace_url + + databricks_workspace_id = module.workspace.workspace_id + databricks_workspace_url = module.workspace.workspace_url google_project = var.workspace_google_project google_region = var.google_region + prefix = var.prefix metastore_name = var.metastore_name catalog_name = var.catalog_name - prefix = var.prefix } \ No newline at end of file diff --git a/examples/gcp-with-psc-exfiltration-protection/variables.tf b/examples/gcp-with-psc-exfiltration-protection/variables.tf index 15365ccf..d4a61fd6 100644 --- a/examples/gcp-with-psc-exfiltration-protection/variables.tf +++ b/examples/gcp-with-psc-exfiltration-protection/variables.tf @@ -10,64 +10,136 @@ variable "google_region" { variable "workspace_google_project" { type = string - description = "Google Cloud project ID related to Databricks workspace" + description = "Google Cloud project ID where the Databricks workspace lives" } variable "spoke_vpc_google_project" { type = string - description = "Google Cloud project ID related to Spoke VPC" + description = "Google Cloud project ID hosting the spoke VPC (often the same as workspace project)" } variable "hub_vpc_google_project" { type = string - description = "Google Cloud project ID related to Hub VPC" + description = "Google Cloud project ID hosting the hub VPC" } variable "is_spoke_vpc_shared" { type = bool - description = "Whether the Spoke VPC is a Shared or a dedicated VPC" + description = "Whether the spoke VPC project hosts a Shared VPC and the workspace project is bound as a service project" } variable "prefix" { type = string - description = "Prefix to use in generated resources name" + description = "Prefix used to name generated resources" } -# For the value of the regional Hive Metastore IP, refer to the Databricks documentation -# Here - https://docs.gcp.databricks.com/en/resources/ip-domain-region.html#addresses-for-default-metastore variable "hive_metastore_ip" { type = string - description = "Value of regional default Hive Metastore IP" + default = null + description = "Regional legacy Hive metastore IP. When set, an egress allow rule (tcp/3306) is created under restricted egress; when null, no rule is created. Workspaces using Unity Catalog (the default) do not need this. Regional IPs: https://docs.databricks.com/gcp/en/resources/ip-domain-region" } variable "hub_vpc_cidr" { type = string - description = "CIDR for Hub VPC" + description = "CIDR for the hub subnet" } variable "spoke_vpc_cidr" { type = string - description = "CIDR for Spoke VPC" + description = "CIDR of the spoke VPC address space (used as source_ranges for the hub ingress firewall)" } -variable "psc_subnet_cidr" { +variable "subnet_cidr" { type = string - description = "CIDR for Spoke VPC" + description = "CIDR for the spoke subnet (must be within spoke_vpc_cidr)" } -variable "tags" { - type = map(string) - description = "Map of tags to add to all resources" - - default = {} +variable "psc_subnet_cidr" { + type = string + description = "CIDR for the dedicated PSC subnet in the spoke VPC" } variable "metastore_name" { type = string - description = "Name to assign to regional metastore" + description = "Name to assign to regional Unity Catalog metastore" } variable "catalog_name" { type = string - description = "Name to assign to default catalog" + description = "Name to assign to default Unity Catalog catalog" +} + +variable "serverless_egress_mode" { + type = string + default = "restricted" + description = "Serverless egress control mode (unmanaged, full, restricted). Default restricted: deny-by-default for serverless, matching this example's classic-compute posture. Requires Enterprise tier" +} + +variable "serverless_allowed_internet_destinations" { + type = list(string) + default = [] + description = "FQDNs serverless workloads may reach (only with serverless_egress_mode=restricted)" +} + +variable "serverless_allowed_storage_destinations" { + type = list(string) + default = [] + description = "GCS bucket names serverless workloads may reach (only with serverless_egress_mode=restricted)" +} + +variable "serverless_egress_enforcement" { + type = string + default = "enforced" + description = "enforced or dry_run (log-only evaluation)" +} + +# === Customer-managed keys (CMEK) ======================================= +variable "cmek_managed_services_key_id" { + type = string + default = null + description = "Cloud KMS key resource ID for managed-services CMEK (control-plane data: notebooks, secrets, queries). Null disables. The principal running Terraform needs cloudkms.cryptoKeys.getIamPolicy and setIamPolicy on the key - Databricks sets the key's IAM policy at workspace creation. Enterprise tier; set at creation only. The key must exist before plan (a key created in the same configuration makes the count unknown and fails plan)" +} + +variable "cmek_storage_key_id" { + type = string + default = null + description = "Cloud KMS key resource ID for workspace-storage CMEK (GCS buckets and GCE persistent disks). Null disables. Same permission and tier requirements as cmek_managed_services_key_id; set at creation only. The key must exist before plan (a key created in the same configuration makes the count unknown and fails plan)" +} + +variable "enable_compliance_security_profile" { + type = bool + default = false + description = "Enable the Compliance Security Profile on the workspace. WARNING: irreversible - CSP cannot be disabled once enabled. Requires enable_enhanced_security_monitoring=true" +} + +variable "compliance_standards" { + type = list(string) + default = [] + description = "Compliance standards for the CSP (e.g. [\"HIPAA\"]). Only meaningful when enable_compliance_security_profile=true" +} + +variable "enable_enhanced_security_monitoring" { + type = bool + default = false + description = "Enable Enhanced Security Monitoring (hardened images, monitoring agents)" +} + +variable "enable_automatic_cluster_update" { + type = bool + default = false + description = "Enable automatic cluster update for the workspace" +} + +variable "ip_access_lists" { + type = list(object({ + label = string + list_type = string + ip_addresses = list(string) + })) + default = [] + description = "Workspace IP access lists. list_type is ALLOW or BLOCK. A non-empty list also flips the enableIpAccessLists workspace conf" + validation { + condition = alltrue([for l in var.ip_access_lists : contains(["ALLOW", "BLOCK"], l.list_type)]) + error_message = "ip_access_lists[*].list_type must be ALLOW or BLOCK." + } } \ No newline at end of file diff --git a/examples/gcp-with-psc-exfiltration-protection/versions.tf b/examples/gcp-with-psc-exfiltration-protection/versions.tf new file mode 100644 index 00000000..9482c733 --- /dev/null +++ b/examples/gcp-with-psc-exfiltration-protection/versions.tf @@ -0,0 +1,17 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { + source = "databricks/databricks" + version = "~> 1.81" + } + google = { + source = "hashicorp/google" + version = "~> 6.17" + } + random = { + source = "hashicorp/random" + version = ">= 3.0" + } + } +} diff --git a/modules/Makefile b/modules/Makefile index 98c80a85..c7d538f8 100644 --- a/modules/Makefile +++ b/modules/Makefile @@ -1,8 +1,14 @@ PROJECTS := $(dir $(wildcard */README.md)) -docs: $(PROJECTS) +docs: $(PROJECTS) gcp-recursive databricks-recursive $(PROJECTS): $(MAKE) -C $@ docs -.PHONY: $(PROJECTS) +gcp-recursive: + $(MAKE) -C gcp docs + +databricks-recursive: + $(MAKE) -C databricks docs + +.PHONY: $(PROJECTS) docs gcp-recursive databricks-recursive diff --git a/modules/databricks/Makefile b/modules/databricks/Makefile new file mode 100644 index 00000000..30b525d1 --- /dev/null +++ b/modules/databricks/Makefile @@ -0,0 +1,8 @@ +PROJECTS := $(dir $(wildcard */README.md)) + +docs: $(PROJECTS) + +$(PROJECTS): + $(MAKE) -C $@ docs + +.PHONY: $(PROJECTS) docs diff --git a/modules/databricks/security-settings/Makefile b/modules/databricks/security-settings/Makefile new file mode 100644 index 00000000..17b32ec8 --- /dev/null +++ b/modules/databricks/security-settings/Makefile @@ -0,0 +1,7 @@ +.PHONY: docs test_docs + +docs: + terraform-docs -c ../../../.terraform-docs.yml . + +test_docs: + terraform-docs -c ../../../.terraform-docs.yml --output-check . diff --git a/modules/databricks/security-settings/README.md b/modules/databricks/security-settings/README.md new file mode 100644 index 00000000..6e6d82cd --- /dev/null +++ b/modules/databricks/security-settings/README.md @@ -0,0 +1,68 @@ +# modules/databricks/security-settings + +Cloud-neutral workspace security settings: Compliance Security Profile, +Enhanced Security Monitoring, automatic cluster update, and IP access lists. +Takes a workspace-level databricks provider from the caller. + +> WARNING: the Compliance Security Profile cannot be disabled once enabled. + +## Usage + +```hcl +module "security_settings" { + source = "github.com/databricks/terraform-databricks-examples//modules/databricks/security-settings" + + providers = { + databricks = databricks.workspace + } + + enable_enhanced_security_monitoring = true + enable_compliance_security_profile = true + compliance_standards = ["HIPAA"] +} +``` + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.5 | +| [databricks](#requirement\_databricks) | >= 1.81.1 | + +## Providers + +| Name | Version | +|------|---------| +| [databricks](#provider\_databricks) | 1.120.0 | +| [terraform](#provider\_terraform) | n/a | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [databricks_automatic_cluster_update_workspace_setting.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/automatic_cluster_update_workspace_setting) | resource | +| [databricks_compliance_security_profile_workspace_setting.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/compliance_security_profile_workspace_setting) | resource | +| [databricks_enhanced_security_monitoring_workspace_setting.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/enhanced_security_monitoring_workspace_setting) | resource | +| [databricks_ip_access_list.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/ip_access_list) | resource | +| [databricks_workspace_conf.enable_ip_access_lists](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/workspace_conf) | resource | +| [terraform_data.preconditions](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [compliance\_standards](#input\_compliance\_standards) | Compliance standards for the CSP (e.g. ["HIPAA"]). Only meaningful when enable\_compliance\_security\_profile=true | `list(string)` | `[]` | no | +| [enable\_automatic\_cluster\_update](#input\_enable\_automatic\_cluster\_update) | Enable automatic cluster update for the workspace | `bool` | `false` | no | +| [enable\_compliance\_security\_profile](#input\_enable\_compliance\_security\_profile) | Enable the Compliance Security Profile on the workspace. WARNING: irreversible - CSP cannot be disabled once enabled. Requires enable\_enhanced\_security\_monitoring=true | `bool` | `false` | no | +| [enable\_enhanced\_security\_monitoring](#input\_enable\_enhanced\_security\_monitoring) | Enable Enhanced Security Monitoring (hardened images, monitoring agents) | `bool` | `false` | no | +| [ip\_access\_lists](#input\_ip\_access\_lists) | Workspace IP access lists. list\_type is ALLOW or BLOCK. A non-empty list also flips the enableIpAccessLists workspace conf | 
list(object({
    label        = string
    list_type    = string
    ip_addresses = list(string)
  }))
| `[]` | no | + +## Outputs + +No outputs. + diff --git a/modules/databricks/security-settings/acu.tf b/modules/databricks/security-settings/acu.tf new file mode 100644 index 00000000..31453f1d --- /dev/null +++ b/modules/databricks/security-settings/acu.tf @@ -0,0 +1,7 @@ +resource "databricks_automatic_cluster_update_workspace_setting" "this" { + count = var.enable_automatic_cluster_update ? 1 : 0 + + automatic_cluster_update_workspace { + enabled = true + } +} diff --git a/modules/databricks/security-settings/csp.tf b/modules/databricks/security-settings/csp.tf new file mode 100644 index 00000000..56cebac3 --- /dev/null +++ b/modules/databricks/security-settings/csp.tf @@ -0,0 +1,8 @@ +resource "databricks_compliance_security_profile_workspace_setting" "this" { + count = var.enable_compliance_security_profile ? 1 : 0 + + compliance_security_profile_workspace { + is_enabled = true + compliance_standards = var.compliance_standards + } +} diff --git a/modules/databricks/security-settings/esm.tf b/modules/databricks/security-settings/esm.tf new file mode 100644 index 00000000..638fa211 --- /dev/null +++ b/modules/databricks/security-settings/esm.tf @@ -0,0 +1,7 @@ +resource "databricks_enhanced_security_monitoring_workspace_setting" "this" { + count = var.enable_enhanced_security_monitoring ? 1 : 0 + + enhanced_security_monitoring_workspace { + is_enabled = true + } +} diff --git a/modules/databricks/security-settings/ip-access-lists.tf b/modules/databricks/security-settings/ip-access-lists.tf new file mode 100644 index 00000000..de24aeec --- /dev/null +++ b/modules/databricks/security-settings/ip-access-lists.tf @@ -0,0 +1,17 @@ +resource "databricks_workspace_conf" "enable_ip_access_lists" { + count = length(var.ip_access_lists) > 0 ? 1 : 0 + + custom_config = { + "enableIpAccessLists" = "true" + } +} + +resource "databricks_ip_access_list" "this" { + for_each = { for l in var.ip_access_lists : l.label => l } + + label = each.value.label + list_type = each.value.list_type + ip_addresses = each.value.ip_addresses + + depends_on = [databricks_workspace_conf.enable_ip_access_lists] +} diff --git a/modules/databricks/security-settings/outputs.tf b/modules/databricks/security-settings/outputs.tf new file mode 100644 index 00000000..8e364848 --- /dev/null +++ b/modules/databricks/security-settings/outputs.tf @@ -0,0 +1 @@ +# This module has no outputs; settings are terminal. diff --git a/modules/databricks/security-settings/preconditions.tf b/modules/databricks/security-settings/preconditions.tf new file mode 100644 index 00000000..f509f2b0 --- /dev/null +++ b/modules/databricks/security-settings/preconditions.tf @@ -0,0 +1,12 @@ +resource "terraform_data" "preconditions" { + lifecycle { + precondition { + condition = !var.enable_compliance_security_profile || var.enable_enhanced_security_monitoring + error_message = "enable_compliance_security_profile=true requires enable_enhanced_security_monitoring=true (CSP builds on ESM)." + } + precondition { + condition = length(var.compliance_standards) == 0 || var.enable_compliance_security_profile + error_message = "compliance_standards requires enable_compliance_security_profile=true." + } + } +} diff --git a/modules/databricks/security-settings/tests/all-enabled/main.tf b/modules/databricks/security-settings/tests/all-enabled/main.tf new file mode 100644 index 00000000..eb0419f3 --- /dev/null +++ b/modules/databricks/security-settings/tests/all-enabled/main.tf @@ -0,0 +1,28 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { source = "databricks/databricks" } + } +} + +provider "databricks" { + host = "https://1234567890123456.7.gcp.databricks.com" + token = "fixture-token" +} + +module "security_settings" { + source = "../.." + + enable_enhanced_security_monitoring = true + enable_compliance_security_profile = true + compliance_standards = ["HIPAA"] + enable_automatic_cluster_update = true + + ip_access_lists = [ + { + label = "corp-vpn" + list_type = "ALLOW" + ip_addresses = ["203.0.113.0/24"] + } + ] +} diff --git a/modules/databricks/security-settings/variables.tf b/modules/databricks/security-settings/variables.tf new file mode 100644 index 00000000..2b985611 --- /dev/null +++ b/modules/databricks/security-settings/variables.tf @@ -0,0 +1,37 @@ +variable "enable_compliance_security_profile" { + type = bool + default = false + description = "Enable the Compliance Security Profile on the workspace. WARNING: irreversible - CSP cannot be disabled once enabled. Requires enable_enhanced_security_monitoring=true" +} + +variable "compliance_standards" { + type = list(string) + default = [] + description = "Compliance standards for the CSP (e.g. [\"HIPAA\"]). Only meaningful when enable_compliance_security_profile=true" +} + +variable "enable_enhanced_security_monitoring" { + type = bool + default = false + description = "Enable Enhanced Security Monitoring (hardened images, monitoring agents)" +} + +variable "enable_automatic_cluster_update" { + type = bool + default = false + description = "Enable automatic cluster update for the workspace" +} + +variable "ip_access_lists" { + type = list(object({ + label = string + list_type = string + ip_addresses = list(string) + })) + default = [] + description = "Workspace IP access lists. list_type is ALLOW or BLOCK. A non-empty list also flips the enableIpAccessLists workspace conf" + validation { + condition = alltrue([for l in var.ip_access_lists : contains(["ALLOW", "BLOCK"], l.list_type)]) + error_message = "ip_access_lists[*].list_type must be ALLOW or BLOCK." + } +} diff --git a/modules/databricks/security-settings/versions.tf b/modules/databricks/security-settings/versions.tf new file mode 100644 index 00000000..11aab3e6 --- /dev/null +++ b/modules/databricks/security-settings/versions.tf @@ -0,0 +1,9 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { + source = "databricks/databricks" + version = ">= 1.81.1" + } + } +} diff --git a/modules/gcp-sa-provisioning/Makefile b/modules/gcp-sa-provisioning/Makefile deleted file mode 100644 index 653039d8..00000000 --- a/modules/gcp-sa-provisioning/Makefile +++ /dev/null @@ -1,7 +0,0 @@ -.PHONY: docs test_docs - -docs: - terraform-docs -c ../../.terraform-docs.yml . - -test_docs: - terraform-docs -c ../../.terraform-docs.yml --output-check . diff --git a/modules/gcp-sa-provisioning/main.tf b/modules/gcp-sa-provisioning/main.tf deleted file mode 100644 index b7f5fc4c..00000000 --- a/modules/gcp-sa-provisioning/main.tf +++ /dev/null @@ -1,144 +0,0 @@ -resource "google_service_account" "sa2" { - account_id = "${var.prefix}-sa2" - display_name = "Service Account for Databricks Provisioning" -} - -output "service_account" { - value = google_service_account.sa2.email - description = "Add this email as a user in the Databricks account console" -} - -data "google_iam_policy" "this" { - binding { - role = "roles/iam.serviceAccountTokenCreator" - members = var.delegate_from - } -} - -resource "google_service_account_iam_policy" "impersonatable" { - service_account_id = google_service_account.sa2.name - policy_data = data.google_iam_policy.this.policy_data -} - -resource "google_project_iam_custom_role" "workspace_creator" { - role_id = "${var.prefix}_workspace_creator" - title = "Databricks Workspace Creator by Aleks" - project = var.google_project - permissions = [ - "compute.globalOperations.get", - "compute.instanceGroups.get", - "compute.instanceGroups.list", - "compute.instances.get", - "compute.instances.list", - "compute.instances.setLabels", - "compute.disks.get", - "compute.disks.setLabels", - "compute.networks.access", - "compute.networks.create", - "compute.networks.delete", - "compute.networks.get", - "compute.networks.updatePolicy", - "compute.networks.use", - "compute.networks.useExternalIp", - "compute.regionOperations.get", - "compute.routers.create", - "compute.routers.delete", - "compute.routers.get", - "compute.routers.update", - "compute.routers.use", - "compute.subnetworks.create", - "compute.subnetworks.delete", - "compute.subnetworks.get", - "compute.subnetworks.expandIpCidrRange", - "compute.subnetworks.getIamPolicy", - "compute.subnetworks.setIamPolicy", - "compute.subnetworks.setPrivateIpGoogleAccess", - "compute.subnetworks.update", - "compute.subnetworks.use", - "compute.subnetworks.useExternalIp", - "container.clusterRoleBindings.create", - "container.clusterRoleBindings.get", - "container.clusterRoles.bind", - "container.clusterRoles.create", - "container.clusterRoles.get", - "container.clusters.create", - "container.clusters.delete", - "container.clusters.get", - "container.clusters.getCredentials", - "container.clusters.list", - "container.clusters.update", - "container.configMaps.create", - "container.configMaps.get", - "container.configMaps.update", - "container.customResourceDefinitions.create", - "container.customResourceDefinitions.get", - "container.customResourceDefinitions.update", - "container.daemonSets.create", - "container.daemonSets.get", - "container.daemonSets.update", - "container.deployments.create", - "container.deployments.get", - "container.deployments.update", - "container.jobs.create", - "container.jobs.get", - "container.jobs.update", - "container.namespaces.create", - "container.namespaces.get", - "container.namespaces.list", - "container.operations.get", - "container.pods.get", - "container.pods.getLogs", - "container.pods.list", - "container.roleBindings.create", - "container.roleBindings.get", - "container.roles.bind", - "container.roles.create", - "container.roles.get", - "container.secrets.create", - "container.secrets.get", - "container.secrets.update", - "container.serviceAccounts.create", - "container.serviceAccounts.get", - "container.services.create", - "container.services.get", - "container.thirdPartyObjects.create", - "container.thirdPartyObjects.delete", - "container.thirdPartyObjects.get", - "container.thirdPartyObjects.update", - "iam.serviceAccounts.getIamPolicy", - "iam.serviceAccounts.setIamPolicy", - "compute.projects.get", - "resourcemanager.projects.get", - "resourcemanager.projects.getIamPolicy", - "storage.buckets.create", - "storage.buckets.delete", - "storage.buckets.get", - "storage.buckets.getIamPolicy", - "storage.buckets.list", - "storage.buckets.setIamPolicy", - "storage.buckets.update", - "storage.hmacKeys.create", - "storage.hmacKeys.get", - "storage.hmacKeys.list", - "storage.hmacKeys.update", - "storage.objects.create", - "storage.objects.delete", - "storage.objects.get", - "storage.objects.getIamPolicy", - "storage.objects.list", - "storage.objects.setIamPolicy", - "storage.objects.update", - "resourcemanager.projects.setIamPolicy", - - ] -} - - -data "google_client_config" "current" {} - -resource "google_project_iam_member" "sa2_can_create_workspaces" { - role = google_project_iam_custom_role.workspace_creator.id - member = "serviceAccount:${google_service_account.sa2.email}" - project = var.google_project - -} diff --git a/modules/gcp-sa-provisioning/outputs.tf b/modules/gcp-sa-provisioning/outputs.tf deleted file mode 100644 index 2ea15177..00000000 --- a/modules/gcp-sa-provisioning/outputs.tf +++ /dev/null @@ -1,4 +0,0 @@ - -output "custom_role_url" { - value = "https://console.cloud.google.com/iam-admin/roles/details/projects%3C${data.google_client_config.current.project}%3Croles%3C${google_project_iam_custom_role.workspace_creator.role_id}" -} diff --git a/modules/gcp-unity-catalog/Makefile b/modules/gcp-unity-catalog/Makefile deleted file mode 100644 index 653039d8..00000000 --- a/modules/gcp-unity-catalog/Makefile +++ /dev/null @@ -1,7 +0,0 @@ -.PHONY: docs test_docs - -docs: - terraform-docs -c ../../.terraform-docs.yml . - -test_docs: - terraform-docs -c ../../.terraform-docs.yml --output-check . diff --git a/modules/gcp-with-psc-exfiltration-protection/Makefile b/modules/gcp-with-psc-exfiltration-protection/Makefile deleted file mode 100644 index 653039d8..00000000 --- a/modules/gcp-with-psc-exfiltration-protection/Makefile +++ /dev/null @@ -1,7 +0,0 @@ -.PHONY: docs test_docs - -docs: - terraform-docs -c ../../.terraform-docs.yml . - -test_docs: - terraform-docs -c ../../.terraform-docs.yml --output-check . diff --git a/modules/gcp-with-psc-exfiltration-protection/README.md b/modules/gcp-with-psc-exfiltration-protection/README.md deleted file mode 100644 index 6f9650de..00000000 --- a/modules/gcp-with-psc-exfiltration-protection/README.md +++ /dev/null @@ -1,139 +0,0 @@ -# Databricks on Google Cloud with Private Service Connect and Hub-Spoke network structure (data exfiltration protection). - -## ⚠️ Prerequisites -To **enable Private Service Connect for your Databricks workspace** on Google Cloud, you must contact your Databricks account team and provide: -- Databricks account ID -- VPC Host Project ID of the **compute plane VPC** for enabling back-end Private Service Connect -- VPC Host Project ID of the **transit VPC** for enabling front-end Private Service Connect -- Workspace region - -This configuration **cannot be completed independently** and requires coordination with your Databricks account team. - -## Overview - -The module includes: -1. Hub-Spoke networking with egress firewall to control all outbound traffic, e.g. to pypi.org. -2. Private Service Connect connection for backend traffic from data plane to control plane. -3. Private Service Connect connection from user client to webapp service. -4. Private Google Access from data plane to DBFS storage. -5. Private Service Connect connection for web-auth traffic. - -## Overall Architecture - -![alt text](images/architecture.png) - -With this deployment, traffic from user client to webapp (notebook UI), backend traffic from data plane to control plane will be through PSC endpoints. This terraform sample will create: -* Hub VPC and its subnet -* Spoke VPC and its subnets -* Peering between Hub and Spoke VPC -* Private Service Connect (PSC) endpoints -* DNS private and peering zones -* Firewall rules for Hub and Spoke VPCs -* Databricks workspace with private link to control plane, user to webapp and private link to DBFS - - -**Note that** the module does not contain the VPC SC implementation. This can be added to increase the security level in the Databricks deployment, providing detailed access level for ingress and egress traffic. -## How to use - -> **Note** -> You can customize this module by adding, deleting or updating the Google Cloud resources to adapt the module to your requirements. -> A deployment example using this module can be found in [examples/gcp-with-psc-exfiltration-protection](../../examples/gcp-with-psc-exfiltration-protection) - -1. Reference this module using one of the different [module source types](https://developer.hashicorp.com/terraform/language/modules/sources) -2. Add `terraform.tfvars` with the information about service principals to be provisioned at account level. - - -## Requirements - -No requirements. - -## Providers - -| Name | Version | -|------|---------| -| [databricks](#provider\_databricks) | n/a | -| [google](#provider\_google) | n/a | -| [random](#provider\_random) | n/a | - -## Modules - -No modules. - -## Resources - -| Name | Type | -|------|------| -| [databricks_mws_networks.databricks_network](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_networks) | resource | -| [databricks_mws_private_access_settings.pas](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_private_access_settings) | resource | -| [databricks_mws_vpc_endpoint.backend_endpoint](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_vpc_endpoint) | resource | -| [databricks_mws_vpc_endpoint.frontend_endpoint](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_vpc_endpoint) | resource | -| [databricks_mws_vpc_endpoint.transit_endpoint](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_vpc_endpoint) | resource | -| [databricks_mws_workspaces.databricks_workspace](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_workspaces) | resource | -| [google_compute_address.backend_pe_ip_address](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_address) | resource | -| [google_compute_address.hub_frontend_pe_ip_address](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_address) | resource | -| [google_compute_address.spoke_frontend_pe_ip_address](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_address) | resource | -| [google_compute_firewall.databricks_workspace_traffic](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource | -| [google_compute_firewall.default_deny_egress](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource | -| [google_compute_firewall.hub_net_traffic](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource | -| [google_compute_firewall.to_databricks_compute_plane](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource | -| [google_compute_firewall.to_databricks_control_plane](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource | -| [google_compute_firewall.to_google_apis](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource | -| [google_compute_firewall.to_managed_hive](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource | -| [google_compute_forwarding_rule.backend_psc_ep](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_forwarding_rule) | resource | -| [google_compute_forwarding_rule.hub_frontend_psc_ep](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_forwarding_rule) | resource | -| [google_compute_forwarding_rule.spoke_frontend_psc_ep](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_forwarding_rule) | resource | -| [google_compute_network.hub_vpc](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_network) | resource | -| [google_compute_network.spoke_vpc](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_network) | resource | -| [google_compute_network_peering.hub_spoke_peering](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_network_peering) | resource | -| [google_compute_network_peering.spoke_hub_peering](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_network_peering) | resource | -| [google_compute_shared_vpc_host_project.host](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_shared_vpc_host_project) | resource | -| [google_compute_shared_vpc_service_project.service](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_shared_vpc_service_project) | resource | -| [google_compute_subnetwork.hub_subnetwork](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_subnetwork) | resource | -| [google_compute_subnetwork.psc_subnetwork](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_subnetwork) | resource | -| [google_compute_subnetwork.spoke_subnetwork](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_subnetwork) | resource | -| [google_dns_managed_zone.gcr_peering_zone](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_managed_zone) | resource | -| [google_dns_managed_zone.gcr_private_zone](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_managed_zone) | resource | -| [google_dns_managed_zone.google_apis_peering_zone](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_managed_zone) | resource | -| [google_dns_managed_zone.google_apis_private_zone](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_managed_zone) | resource | -| [google_dns_managed_zone.hub_private_zone](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_managed_zone) | resource | -| [google_dns_managed_zone.pkg_dev_peering_zone](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_managed_zone) | resource | -| [google_dns_managed_zone.pkg_dev_private_zone](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_managed_zone) | resource | -| [google_dns_managed_zone.spoke_private_zone](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_managed_zone) | resource | -| [google_dns_record_set.gcr_a](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | -| [google_dns_record_set.gcr_cname](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | -| [google_dns_record_set.hub_workspace_dp](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | -| [google_dns_record_set.hub_workspace_psc_auth](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | -| [google_dns_record_set.hub_workspace_url](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | -| [google_dns_record_set.pkg_dev_a](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | -| [google_dns_record_set.pkg_dev_cname](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | -| [google_dns_record_set.restricted_apis_a](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | -| [google_dns_record_set.restricted_apis_cname](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | -| [google_dns_record_set.spoke_relay](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | -| [google_dns_record_set.spoke_workspace_dp](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | -| [google_dns_record_set.spoke_workspace_url](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | -| [random_string.suffix](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/string) | resource | - -## Inputs - -| Name | Description | Type | Default | Required | -|------|-------------|------|---------|:--------:| -| [databricks\_account\_id](#input\_databricks\_account\_id) | Databricks Account ID | `string` | n/a | yes | -| [google\_region](#input\_google\_region) | Google Cloud region where the resources will be created | `string` | n/a | yes | -| [hive\_metastore\_ip](#input\_hive\_metastore\_ip) | Value of regional default Hive Metastore IP | `string` | n/a | yes | -| [hub\_vpc\_cidr](#input\_hub\_vpc\_cidr) | CIDR for Hub VPC | `string` | n/a | yes | -| [hub\_vpc\_google\_project](#input\_hub\_vpc\_google\_project) | Google Cloud project ID related to Hub VPC | `string` | n/a | yes | -| [is\_spoke\_vpc\_shared](#input\_is\_spoke\_vpc\_shared) | Whether the Spoke VPC is a Shared or a dedicated VPC | `bool` | n/a | yes | -| [prefix](#input\_prefix) | Prefix to use in generated resources name | `string` | n/a | yes | -| [psc\_subnet\_cidr](#input\_psc\_subnet\_cidr) | CIDR for Spoke VPC | `string` | n/a | yes | -| [spoke\_vpc\_cidr](#input\_spoke\_vpc\_cidr) | CIDR for Spoke VPC | `string` | n/a | yes | -| [spoke\_vpc\_google\_project](#input\_spoke\_vpc\_google\_project) | Google Cloud project ID related to Spoke VPC | `string` | n/a | yes | -| [tags](#input\_tags) | Map of tags to add to all resources | `map(string)` | n/a | yes | -| [workspace\_google\_project](#input\_workspace\_google\_project) | Google Cloud project ID related to Databricks workspace | `string` | n/a | yes | - -## Outputs - -| Name | Description | -|------|-------------| -| [workspace\_id](#output\_workspace\_id) | The Databricks workspace ID | -| [workspace\_url](#output\_workspace\_url) | The workspace URL which is of the format '{workspaceId}.{random}.gcp.databricks.com' | - \ No newline at end of file diff --git a/modules/gcp-with-psc-exfiltration-protection/databricks-cloud-resources.tf b/modules/gcp-with-psc-exfiltration-protection/databricks-cloud-resources.tf deleted file mode 100644 index f0a355e0..00000000 --- a/modules/gcp-with-psc-exfiltration-protection/databricks-cloud-resources.tf +++ /dev/null @@ -1,86 +0,0 @@ -################################################### -# Databricks VPC Endpoints & Network Configuration -################################################### - -# ================================================ -# Private Service Connect Endpoint Configurations -# ================================================ - -# Registers a transit VPC endpoint for hub network connectivity -resource "databricks_mws_vpc_endpoint" "transit_endpoint" { - depends_on = [google_compute_forwarding_rule.backend_psc_ep] - - vpc_endpoint_name = "${var.prefix}-hub-ep-${random_string.suffix.result}" - account_id = var.databricks_account_id - - # GCP-specific PSC configuration for hub network - gcp_vpc_endpoint_info { - project_id = var.hub_vpc_google_project - psc_endpoint_name = google_compute_forwarding_rule.hub_frontend_psc_ep.name - endpoint_region = var.google_region - } -} - -# Registers frontend workspace VPC endpoint for user-facing access -resource "databricks_mws_vpc_endpoint" "frontend_endpoint" { - depends_on = [google_compute_forwarding_rule.backend_psc_ep] - - vpc_endpoint_name = "${var.prefix}-ws-ep-${random_string.suffix.result}" - account_id = var.databricks_account_id - - # GCP-specific PSC configuration for spoke workspace - gcp_vpc_endpoint_info { - project_id = var.spoke_vpc_google_project - psc_endpoint_name = google_compute_forwarding_rule.spoke_frontend_psc_ep.name - endpoint_region = var.google_region - } -} - -# Registers backend SCC (Secure Cluster Connectivity) endpoint -resource "databricks_mws_vpc_endpoint" "backend_endpoint" { - depends_on = [google_compute_forwarding_rule.spoke_frontend_psc_ep] - - vpc_endpoint_name = "${var.prefix}-scc-ep-${random_string.suffix.result}" - account_id = var.databricks_account_id - - # GCP-specific PSC configuration for backend connectivity - gcp_vpc_endpoint_info { - project_id = var.spoke_vpc_google_project - psc_endpoint_name = google_compute_forwarding_rule.backend_psc_ep.name - endpoint_region = var.google_region - } -} - -# ================================================ -# Network Configuration for Databricks Workspace -# ================================================ - -resource "databricks_mws_networks" "databricks_network" { - network_name = "${var.prefix}-ntw-${random_string.suffix.result}" - account_id = var.databricks_account_id - - # GCP network infrastructure details - gcp_network_info { - network_project_id = var.spoke_vpc_google_project - vpc_id = google_compute_network.spoke_vpc.name - subnet_id = google_compute_subnetwork.spoke_subnetwork.name - subnet_region = var.google_region - } - - # PrivateLink endpoint associations - vpc_endpoints { - dataplane_relay = [databricks_mws_vpc_endpoint.backend_endpoint.vpc_endpoint_id] # SCC connectivity - rest_api = [databricks_mws_vpc_endpoint.frontend_endpoint.vpc_endpoint_id] # Workspace API access - } -} - -# ================================================ -# Private Access Configuration -# ================================================ - -resource "databricks_mws_private_access_settings" "pas" { - private_access_settings_name = "${var.prefix}-pas-${random_string.suffix.result}" - region = var.google_region - public_access_enabled = false # Block public internet access - private_access_level = "ACCOUNT" # Apply to entire Databricks account -} diff --git a/modules/gcp-with-psc-exfiltration-protection/dns-hub.tf b/modules/gcp-with-psc-exfiltration-protection/dns-hub.tf deleted file mode 100644 index b0eca334..00000000 --- a/modules/gcp-with-psc-exfiltration-protection/dns-hub.tf +++ /dev/null @@ -1,214 +0,0 @@ -######################################### -# Databricks Private DNS Configuration # -######################################### - -# Create a private DNS zone for Databricks PSC management -resource "google_dns_managed_zone" "hub_private_zone" { - name = "${var.prefix}-hub-gcp-databricks-com" - project = var.hub_vpc_google_project - dns_name = "gcp.databricks.com." - description = "Private DNS zone for Databricks PSC management" - visibility = "private" - - # Restrict visibility to the hub VPC network - private_visibility_config { - networks { - network_url = google_compute_network.hub_vpc.id - } - } -} - -# DNS A record for the Databricks workspace URL -resource "google_dns_record_set" "hub_workspace_url" { - name = "${local.workspace_dns_id}.${google_dns_managed_zone.hub_private_zone.dns_name}" - project = var.hub_vpc_google_project - managed_zone = google_dns_managed_zone.hub_private_zone.name - type = "A" - ttl = 300 - - # Points to the Databricks frontend Private Endpoint IP address - rrdatas = [ - google_compute_address.hub_frontend_pe_ip_address.address - ] -} - -# DNS A record for the Databricks PSC authentication endpoint -resource "google_dns_record_set" "hub_workspace_psc_auth" { - name = "${var.google_region}.psc-auth.${google_dns_managed_zone.hub_private_zone.dns_name}" - project = var.hub_vpc_google_project - managed_zone = google_dns_managed_zone.hub_private_zone.name - type = "A" - ttl = 300 - - # Points to the same frontend Private Endpoint IP - rrdatas = [ - google_compute_address.hub_frontend_pe_ip_address.address - ] -} - -# DNS A record for the Databricks dataplane endpoint -resource "google_dns_record_set" "hub_workspace_dp" { - name = "dp-${local.workspace_dns_id}.${google_dns_managed_zone.hub_private_zone.dns_name}" - project = var.hub_vpc_google_project - managed_zone = google_dns_managed_zone.hub_private_zone.name - type = "A" - ttl = 300 - - # Points to the same frontend Private Endpoint IP - rrdatas = [ - google_compute_address.hub_frontend_pe_ip_address.address - ] -} - -############################################# -# Google Container Registry Private DNS Zone # -############################################# - -# Create a private DNS zone for GCR (gcr.io) -resource "google_dns_managed_zone" "gcr_private_zone" { - name = "${var.prefix}-gcr-io" - project = var.hub_vpc_google_project - dns_name = "gcr.io." - description = "Private DNS zone for GCR private resolution" - visibility = "private" - - # Restrict visibility to the hub VPC network - private_visibility_config { - networks { - network_url = google_compute_network.hub_vpc.id - } - } -} - -# Wildcard CNAME record for all subdomains of gcr.io -resource "google_dns_record_set" "gcr_cname" { - name = "*.${google_dns_managed_zone.gcr_private_zone.dns_name}" - project = var.hub_vpc_google_project - managed_zone = google_dns_managed_zone.gcr_private_zone.name - type = "CNAME" - ttl = 300 - - # All subdomains point to gcr.io - rrdatas = [ - "gcr.io." - ] -} - -# A record for gcr.io pointing to Google IPs for private access -resource "google_dns_record_set" "gcr_a" { - name = google_dns_managed_zone.gcr_private_zone.dns_name - project = var.hub_vpc_google_project - managed_zone = google_dns_managed_zone.gcr_private_zone.name - type = "A" - ttl = 300 - - # Official Google IPs for gcr.io - rrdatas = [ - "199.36.153.8", - "199.36.153.9", - "199.36.153.10", - "199.36.153.11" - ] -} - -################################## -# Google APIs Private DNS Zone # -################################## - -# Create a private DNS zone for Google APIs (googleapis.com) -resource "google_dns_managed_zone" "google_apis_private_zone" { - name = "${var.prefix}-google-apis" - project = var.hub_vpc_google_project - dns_name = "googleapis.com." - description = "Private DNS zone for Google APIs resolution" - visibility = "private" - - # Restrict visibility to the hub VPC network - private_visibility_config { - networks { - network_url = google_compute_network.hub_vpc.id - } - } -} - -# Wildcard CNAME record for all subdomains of googleapis.com -resource "google_dns_record_set" "restricted_apis_cname" { - name = "*.${google_dns_managed_zone.google_apis_private_zone.dns_name}" - project = var.hub_vpc_google_project - managed_zone = google_dns_managed_zone.google_apis_private_zone.name - type = "CNAME" - ttl = 300 - - # All subdomains point to restricted.googleapis.com - rrdatas = [ - "restricted.googleapis.com." - ] -} - -# A record for restricted.googleapis.com pointing to Google IPs for private access -resource "google_dns_record_set" "restricted_apis_a" { - name = "restricted.${google_dns_managed_zone.google_apis_private_zone.dns_name}" - project = var.hub_vpc_google_project - managed_zone = google_dns_managed_zone.google_apis_private_zone.name - type = "A" - ttl = 300 - - # Official Google IPs for restricted.googleapis.com - rrdatas = [ - "199.36.153.4", - "199.36.153.5", - "199.36.153.6", - "199.36.153.7" - ] -} - -################################## -# Go Packages Private DNS Zone # -################################## - -# Create a private DNS zone for Go Packages (pkg.dev) -resource "google_dns_managed_zone" "pkg_dev_private_zone" { - name = "${var.prefix}-pkg-dev" - project = var.hub_vpc_google_project - dns_name = "pkg.dev." - description = "Private DNS zone for Go Packages resolution" - visibility = "private" - - # Restrict visibility to the hub VPC network - private_visibility_config { - networks { - network_url = google_compute_network.hub_vpc.id - } - } -} - -# Wildcard CNAME record for all subdomains of pkg.dev -resource "google_dns_record_set" "pkg_dev_cname" { - name = "*.${google_dns_managed_zone.pkg_dev_private_zone.dns_name}" - project = var.hub_vpc_google_project - managed_zone = google_dns_managed_zone.pkg_dev_private_zone.name - type = "CNAME" - ttl = 300 - - # All subdomains point to pkg.dev - rrdatas = [ - "pkg.dev." - ] -} - -# A record for pkg.dev pointing to Google IPs for private access -resource "google_dns_record_set" "pkg_dev_a" { - name = google_dns_managed_zone.pkg_dev_private_zone.dns_name - project = var.hub_vpc_google_project - managed_zone = google_dns_managed_zone.pkg_dev_private_zone.name - type = "A" - ttl = 300 - - # Official Google IPs for pkg.dev - rrdatas = [ - "199.36.153.8", - "199.36.153.9", - "199.36.153.10", - "199.36.153.11" - ] -} diff --git a/modules/gcp-with-psc-exfiltration-protection/dns-spoke.tf b/modules/gcp-with-psc-exfiltration-protection/dns-spoke.tf deleted file mode 100644 index 799cd81a..00000000 --- a/modules/gcp-with-psc-exfiltration-protection/dns-spoke.tf +++ /dev/null @@ -1,135 +0,0 @@ -############################################# -# Databricks Private DNS Zone (Spoke VPC) # -############################################# - -# Creates a private DNS managed zone for Databricks PSC endpoints -# This zone is only visible within the spoke VPC network -resource "google_dns_managed_zone" "spoke_private_zone" { - name = "${var.prefix}-spoke-gcp-databricks-com" - project = var.spoke_vpc_google_project - dns_name = "gcp.databricks.com." - description = "Private DNS zone for Databricks PSC management" - visibility = "private" - - # Restricts DNS zone visibility to the spoke VPC - private_visibility_config { - networks { - network_url = google_compute_network.spoke_vpc.id - } - } -} - -# Creates an A record for the Databricks workspace endpoint in the spoke VPC -resource "google_dns_record_set" "spoke_workspace_url" { - name = "${local.workspace_dns_id}.${google_dns_managed_zone.spoke_private_zone.dns_name}" - project = var.spoke_vpc_google_project - managed_zone = google_dns_managed_zone.spoke_private_zone.name - type = "A" - ttl = 300 - - # Points to the Databricks frontend Private Endpoint IP in the spoke VPC - rrdatas = [ - google_compute_address.spoke_frontend_pe_ip_address.address - ] -} - -# Creates an A record for the Databricks dataplane endpoint in the spoke VPC -resource "google_dns_record_set" "spoke_workspace_dp" { - name = "dp-${local.workspace_dns_id}.${google_dns_managed_zone.spoke_private_zone.dns_name}" - project = var.spoke_vpc_google_project - managed_zone = google_dns_managed_zone.spoke_private_zone.name - type = "A" - ttl = 300 - - # Points to the Databricks frontend Private Endpoint IP in the spoke VPC - rrdatas = [ - google_compute_address.spoke_frontend_pe_ip_address.address - ] -} - -# Creates an A record for the Databricks relay/tunnel endpoint in the spoke VPC -resource "google_dns_record_set" "spoke_relay" { - name = "tunnel.${var.google_region}.${google_dns_managed_zone.spoke_private_zone.dns_name}" - project = var.spoke_vpc_google_project - managed_zone = google_dns_managed_zone.spoke_private_zone.name - type = "A" - ttl = 300 - - # Points to the backend Private Endpoint IP (used for relay/tunnel) - rrdatas = [ - google_compute_address.backend_pe_ip_address.address - ] -} - -########################################################## -# Peering DNS Zones for Hub-Spoke Shared Service Access # -########################################################## - -# The following managed zones provide private DNS for Google services (GCR, Google APIs, Go Packages) -# and are peered to the hub VPC for shared DNS resolution across VPCs. - -# Google Container Registry (GCR) private peering zone -resource "google_dns_managed_zone" "gcr_peering_zone" { - name = "${var.prefix}-peering-gcr" - project = var.spoke_vpc_google_project - dns_name = "gcr.io." - description = "Peering DNS zone for GCR private resolution" - visibility = "private" - - private_visibility_config { - networks { - network_url = google_compute_network.spoke_vpc.id - } - } - - # Peers this DNS zone with the hub VPC to allow DNS resolution from the hub - peering_config { - target_network { - network_url = google_compute_network.hub_vpc.id - } - } -} - -# Google APIs private peering zone -resource "google_dns_managed_zone" "google_apis_peering_zone" { - name = "${var.prefix}-peering-google-apis" - project = var.spoke_vpc_google_project - dns_name = "googleapis.com." - description = "Private DNS zone for Google APIs resolution" - visibility = "private" - - private_visibility_config { - networks { - network_url = google_compute_network.spoke_vpc.id - } - } - - # Peers this DNS zone with the hub VPC to allow DNS resolution from the hub - peering_config { - target_network { - network_url = google_compute_network.hub_vpc.id - } - } -} - -# Go Packages (pkg.dev) private peering zone -resource "google_dns_managed_zone" "pkg_dev_peering_zone" { - name = "${var.prefix}-peering-pkg-dev" - project = var.spoke_vpc_google_project - dns_name = "pkg.dev." - description = "Private DNS zone for Go Packages resolution" - visibility = "private" - - private_visibility_config { - networks { - network_url = google_compute_network.spoke_vpc.id - } - } - - # Peers this DNS zone with the hub VPC to allow DNS resolution from the hub - peering_config { - target_network { - network_url = google_compute_network.hub_vpc.id - } - } -} diff --git a/modules/gcp-with-psc-exfiltration-protection/firewall-hub.tf b/modules/gcp-with-psc-exfiltration-protection/firewall-hub.tf deleted file mode 100644 index a1563a2e..00000000 --- a/modules/gcp-with-psc-exfiltration-protection/firewall-hub.tf +++ /dev/null @@ -1,21 +0,0 @@ -# ========================================================== -# Google Cloud VPC Firewall Rule: Hub Network Ingress Traffic -# ========================================================== - -resource "google_compute_firewall" "hub_net_traffic" { - name = "${google_compute_network.hub_vpc.name}-ingress" - - project = var.hub_vpc_google_project - network = google_compute_network.hub_vpc.self_link - - direction = "INGRESS" - priority = 1000 - destination_ranges = [] - # The source IP range(s) allowed by this rule (CIDR format) - # Only traffic originating from the spoke VPC's CIDR block will be allowed - source_ranges = [var.spoke_vpc_cidr] - - allow { - protocol = "all" - } -} diff --git a/modules/gcp-with-psc-exfiltration-protection/firewall-spoke.tf b/modules/gcp-with-psc-exfiltration-protection/firewall-spoke.tf deleted file mode 100644 index a44c69a6..00000000 --- a/modules/gcp-with-psc-exfiltration-protection/firewall-spoke.tf +++ /dev/null @@ -1,112 +0,0 @@ -############################################################# -# Google Cloud Firewall Rules for Databricks Spoke Network # -############################################################# - -# ========================================================== -# Default Egress Deny Rule (Catch-All Block) -# ========================================================== - -resource "google_compute_firewall" "default_deny_egress" { - name = "${google_compute_network.spoke_vpc.name}-default-deny-egress" - project = var.spoke_vpc_google_project - network = google_compute_network.spoke_vpc.self_link - - direction = "EGRESS" - priority = 1100 # Higher priority than allow rules - destination_ranges = ["0.0.0.0/0"] # Block all external destinations - source_ranges = [] - - deny { protocol = "all" } # Explicit deny all outbound traffic -} - -# ========================================================== -# Essential Service Allow Rules -# ========================================================== - -# Allows outbound traffic to Google APIs and services -resource "google_compute_firewall" "to_google_apis" { - name = "${google_compute_network.spoke_vpc.name}-to-google-apis" - project = var.spoke_vpc_google_project - network = google_compute_network.spoke_vpc.self_link - - direction = "EGRESS" - priority = 1000 # Lower priority than deny rule - destination_ranges = [ - "199.36.153.4/30", # Restricted Google APIs - "199.36.153.8/30", # GCR/GCS endpoints - "34.126.0.0/18" # Additional Google service IPs - ] - - allow { protocol = "all" } # Full protocol access to these IPs -} - -# Allows control plane communication for Databricks -resource "google_compute_firewall" "to_databricks_control_plane" { - name = "${google_compute_network.spoke_vpc.name}-to-databricks-control-plane" - project = var.spoke_vpc_google_project - network = google_compute_network.spoke_vpc.self_link - - direction = "EGRESS" - priority = 1000 - destination_ranges = [ - "${google_compute_forwarding_rule.backend_psc_ep.ip_address}/32", # SCC endpoint - "${google_compute_forwarding_rule.spoke_frontend_psc_ep.ip_address}/32" # Frontend endpoint - ] - - allow { - protocol = "tcp" - ports = ["443"] # HTTPS only - } -} - -# ========================================================== -# Managed Hive Metastore Access (Conditional) -# ========================================================== - -resource "google_compute_firewall" "to_managed_hive" { - name = "${google_compute_network.spoke_vpc.name}-to-${var.google_region}-managed-hive" - project = var.spoke_vpc_google_project - network = google_compute_network.spoke_vpc.self_link - - direction = "EGRESS" - priority = 1000 - destination_ranges = ["${var.hive_metastore_ip}/32"] # Metastore-specific IP - - allow { - protocol = "tcp" - ports = ["3306"] # MySQL port - } -} - -# ========================================================== -# Internal Workspace Communication -# ========================================================== - -resource "google_compute_firewall" "databricks_workspace_traffic" { - name = "${google_compute_network.spoke_vpc.name}-${databricks_mws_workspaces.databricks_workspace.workspace_id}-ingress" - project = var.spoke_vpc_google_project - network = google_compute_network.spoke_vpc.self_link - - direction = "INGRESS" - priority = 1000 - source_ranges = [var.spoke_vpc_cidr] # Internal VPC traffic - target_tags = ["databricks-${databricks_mws_workspaces.databricks_workspace.workspace_id}"] # Workspace-specific instances - - allow { protocol = "all" } # Full internal access -} - -resource "google_compute_firewall" "to_databricks_compute_plane" { - name = "${google_compute_network.spoke_vpc.name}-to-databricks-compute-plane" - project = var.spoke_vpc_google_project - network = google_compute_network.spoke_vpc.self_link - - direction = "EGRESS" - priority = 1000 - destination_ranges = [ - var.spoke_vpc_cidr - ] - - allow { - protocol = "all" - } -} \ No newline at end of file diff --git a/modules/gcp-with-psc-exfiltration-protection/images/architecture.png b/modules/gcp-with-psc-exfiltration-protection/images/architecture.png deleted file mode 100644 index 9b245904..00000000 Binary files a/modules/gcp-with-psc-exfiltration-protection/images/architecture.png and /dev/null differ diff --git a/modules/gcp-with-psc-exfiltration-protection/outputs.tf b/modules/gcp-with-psc-exfiltration-protection/outputs.tf deleted file mode 100644 index 27983846..00000000 --- a/modules/gcp-with-psc-exfiltration-protection/outputs.tf +++ /dev/null @@ -1,10 +0,0 @@ - -output "workspace_url" { - value = databricks_mws_workspaces.databricks_workspace.workspace_url - description = "The workspace URL which is of the format '{workspaceId}.{random}.gcp.databricks.com'" -} - -output "workspace_id" { - description = "The Databricks workspace ID" - value = databricks_mws_workspaces.databricks_workspace.workspace_id -} \ No newline at end of file diff --git a/modules/gcp-with-psc-exfiltration-protection/psc.tf b/modules/gcp-with-psc-exfiltration-protection/psc.tf deleted file mode 100644 index e64b5436..00000000 --- a/modules/gcp-with-psc-exfiltration-protection/psc.tf +++ /dev/null @@ -1,75 +0,0 @@ -######################################################### -# Private Service Connect (PSC) Internal Endpoints Setup -######################################################### - -# ---------------------------------------------------------------- -# Secure Cluster Connectivity (SCC) PSC Endpoint (Spoke VPC) -# ---------------------------------------------------------------- - -# Reserves an internal IP address for the backend (SCC) PSC endpoint in the spoke VPC -resource "google_compute_address" "backend_pe_ip_address" { - name = "${var.prefix}-psc-scc-ip-${random_string.suffix.result}" - project = var.spoke_vpc_google_project - region = var.google_region - subnetwork = google_compute_subnetwork.psc_subnetwork.name - address_type = "INTERNAL" -} - -# Creates a forwarding rule to map the reserved IP to the SCC PSC service attachment -resource "google_compute_forwarding_rule" "backend_psc_ep" { - name = "${var.prefix}-psc-scc-ep-${random_string.suffix.result}" - project = var.spoke_vpc_google_project - region = var.google_region - network = google_compute_network.spoke_vpc.id - ip_address = google_compute_address.backend_pe_ip_address.id - target = local.google_backend_psc_targets[var.google_region] - load_balancing_scheme = "" # Must be set to "" for service attachment targets -} - -# ---------------------------------------------------------------- -# Workspace Frontend PSC Endpoint (Spoke VPC) -# ---------------------------------------------------------------- - -# Reserves an internal IP address for the workspace frontend PSC endpoint in the spoke VPC -resource "google_compute_address" "spoke_frontend_pe_ip_address" { - name = "${var.prefix}-psc-ws-ip-${random_string.suffix.result}" - project = var.spoke_vpc_google_project - region = var.google_region - subnetwork = google_compute_subnetwork.psc_subnetwork.name - address_type = "INTERNAL" -} - -# Creates a forwarding rule to map the reserved IP to the workspace frontend PSC service attachment -resource "google_compute_forwarding_rule" "spoke_frontend_psc_ep" { - name = "${var.prefix}-psc-ws-ep-${random_string.suffix.result}" - project = var.spoke_vpc_google_project - region = var.google_region - network = google_compute_network.spoke_vpc.id - ip_address = google_compute_address.spoke_frontend_pe_ip_address.id - target = local.google_frontend_psc_targets[var.google_region] - load_balancing_scheme = "" # Must be set to "" for service attachment targets -} - -# ---------------------------------------------------------------- -# Workspace Frontend PSC Endpoint (Hub VPC) -# ---------------------------------------------------------------- - -# Reserves an internal IP address for the workspace frontend PSC endpoint in the hub VPC -resource "google_compute_address" "hub_frontend_pe_ip_address" { - name = "${var.prefix}-hub-psc-ws-ip-${random_string.suffix.result}" - project = var.hub_vpc_google_project - region = var.google_region - subnetwork = google_compute_subnetwork.hub_subnetwork.name - address_type = "INTERNAL" -} - -# Creates a forwarding rule to map the reserved IP to the workspace frontend PSC service attachment in the hub VPC -resource "google_compute_forwarding_rule" "hub_frontend_psc_ep" { - name = "${var.prefix}-hub-psc-ws-ep-${random_string.suffix.result}" - project = var.hub_vpc_google_project - region = var.google_region - network = google_compute_network.hub_vpc.id - ip_address = google_compute_address.hub_frontend_pe_ip_address.id - target = local.google_frontend_psc_targets[var.google_region] - load_balancing_scheme = "" # Must be set to "" for service attachment targets -} diff --git a/modules/gcp-with-psc-exfiltration-protection/terraform.tf b/modules/gcp-with-psc-exfiltration-protection/terraform.tf deleted file mode 100644 index 688f0fbd..00000000 --- a/modules/gcp-with-psc-exfiltration-protection/terraform.tf +++ /dev/null @@ -1,13 +0,0 @@ -terraform { - required_providers { - databricks = { - source = "databricks/databricks" - } - google = { - source = "hashicorp/google" - } - random = { - source = "hashicorp/random" - } - } -} \ No newline at end of file diff --git a/modules/gcp-with-psc-exfiltration-protection/variables.tf b/modules/gcp-with-psc-exfiltration-protection/variables.tf deleted file mode 100644 index cd96e520..00000000 --- a/modules/gcp-with-psc-exfiltration-protection/variables.tf +++ /dev/null @@ -1,62 +0,0 @@ -variable "databricks_account_id" { - type = string - description = "Databricks Account ID" -} - -variable "google_region" { - type = string - description = "Google Cloud region where the resources will be created" -} - -variable "workspace_google_project" { - type = string - description = "Google Cloud project ID related to Databricks workspace" -} - -variable "spoke_vpc_google_project" { - type = string - description = "Google Cloud project ID related to Spoke VPC" -} - -variable "hub_vpc_google_project" { - type = string - description = "Google Cloud project ID related to Hub VPC" -} - -variable "is_spoke_vpc_shared" { - type = bool - description = "Whether the Spoke VPC is a Shared or a dedicated VPC" -} - -variable "prefix" { - type = string - description = "Prefix to use in generated resources name" -} - -# For the value of the regional Hive Metastore IP, refer to the Databricks documentation -# Here - https://docs.gcp.databricks.com/en/resources/ip-domain-region.html#addresses-for-default-metastore -variable "hive_metastore_ip" { - type = string - description = "Value of regional default Hive Metastore IP" -} - -variable "hub_vpc_cidr" { - type = string - description = "CIDR for Hub VPC" -} - -variable "spoke_vpc_cidr" { - type = string - description = "CIDR for Spoke VPC" -} - -variable "psc_subnet_cidr" { - type = string - description = "CIDR for Spoke VPC" -} - -variable "tags" { - description = "Map of tags to add to all resources" - type = map(string) -} - diff --git a/modules/gcp-with-psc-exfiltration-protection/vpc.tf b/modules/gcp-with-psc-exfiltration-protection/vpc.tf deleted file mode 100644 index 113c873d..00000000 --- a/modules/gcp-with-psc-exfiltration-protection/vpc.tf +++ /dev/null @@ -1,90 +0,0 @@ -######################################################### -# Hub & Spoke Network Infrastructure Configuration -######################################################### - -# ======================================================= -# VPC Networks -# ======================================================= - -# Spoke VPC for Databricks workspace and workloads -resource "google_compute_network" "spoke_vpc" { - name = "${var.prefix}-spoke-vpc-${random_string.suffix.result}" - project = var.spoke_vpc_google_project - auto_create_subnetworks = false # Manual subnet configuration - routing_mode = "GLOBAL" # Global routing for hybrid connectivity - bgp_best_path_selection_mode = "STANDARD" -} - -# Hub VPC for centralized networking services -resource "google_compute_network" "hub_vpc" { - name = "${var.prefix}-hub-vpc-${random_string.suffix.result}" - project = var.hub_vpc_google_project - auto_create_subnetworks = false - routing_mode = "GLOBAL" -} - -# ======================================================= -# Subnetwork Configuration -# ======================================================= - -# Primary spoke subnet for general workloads -resource "google_compute_subnetwork" "spoke_subnetwork" { - name = "${var.prefix}-spoke-subnet-${random_string.suffix.result}" - project = var.spoke_vpc_google_project - network = google_compute_network.spoke_vpc.id - region = var.google_region - ip_cidr_range = var.spoke_vpc_cidr - private_ip_google_access = true # Enables Private Google Access -} - -# Dedicated PSC subnet for Private Service Connect endpoints -resource "google_compute_subnetwork" "psc_subnetwork" { - name = "${var.prefix}-spoke-psc-subnet-${random_string.suffix.result}" - project = var.spoke_vpc_google_project - network = google_compute_network.spoke_vpc.id - region = var.google_region - ip_cidr_range = var.psc_subnet_cidr - private_ip_google_access = true -} - -# Hub subnet for shared services -resource "google_compute_subnetwork" "hub_subnetwork" { - name = "${var.prefix}-hub-subnet-${random_string.suffix.result}" - project = var.hub_vpc_google_project - network = google_compute_network.hub_vpc.id - region = var.google_region - ip_cidr_range = var.hub_vpc_cidr - private_ip_google_access = true -} - -# ======================================================= -# Network Peering Configuration -# ======================================================= - -# Bidirectional peering between hub and spoke VPCs -resource "google_compute_network_peering" "hub_spoke_peering" { - name = "${var.prefix}-hub-spoke-peering-${random_string.suffix.result}" - network = google_compute_network.hub_vpc.self_link - peer_network = google_compute_network.spoke_vpc.self_link -} - -resource "google_compute_network_peering" "spoke_hub_peering" { - name = "${var.prefix}-spoke-hub-peering-${random_string.suffix.result}" - network = google_compute_network.spoke_vpc.self_link - peer_network = google_compute_network.hub_vpc.self_link -} - -# ======================================================= -# Shared VPC Configuration (Conditional) -# ======================================================= - -resource "google_compute_shared_vpc_host_project" "host" { - count = var.workspace_google_project != var.spoke_vpc_google_project && var.is_spoke_vpc_shared ? 1 : 0 - project = var.spoke_vpc_google_project -} - -resource "google_compute_shared_vpc_service_project" "service" { - count = var.workspace_google_project != var.spoke_vpc_google_project && var.is_spoke_vpc_shared ? 1 : 0 - host_project = google_compute_shared_vpc_host_project.host[0].project - service_project = var.workspace_google_project -} diff --git a/modules/gcp-with-psc-exfiltration-protection/workspace.tf b/modules/gcp-with-psc-exfiltration-protection/workspace.tf deleted file mode 100644 index 44dc510e..00000000 --- a/modules/gcp-with-psc-exfiltration-protection/workspace.tf +++ /dev/null @@ -1,22 +0,0 @@ -######################################################### -# Databricks Workspace Configuration -######################################################### - -resource "databricks_mws_workspaces" "databricks_workspace" { - workspace_name = "${var.prefix}-ws-${random_string.suffix.result}" - - # Databricks account and cloud provider details - account_id = var.databricks_account_id - location = var.google_region # GCP region for workspace deployment - - # GCP project hosting workspace resources - cloud_resource_container { - gcp { - project_id = var.workspace_google_project - } - } - - # Network and security configurations - private_access_settings_id = databricks_mws_private_access_settings.pas.private_access_settings_id # Private access enforcement - network_id = databricks_mws_networks.databricks_network.network_id # Associated VPC network -} diff --git a/modules/gcp-workspace-basic/Makefile b/modules/gcp-workspace-basic/Makefile deleted file mode 100644 index 653039d8..00000000 --- a/modules/gcp-workspace-basic/Makefile +++ /dev/null @@ -1,7 +0,0 @@ -.PHONY: docs test_docs - -docs: - terraform-docs -c ../../.terraform-docs.yml . - -test_docs: - terraform-docs -c ../../.terraform-docs.yml --output-check . diff --git a/modules/gcp-workspace-basic/README.md b/modules/gcp-workspace-basic/README.md deleted file mode 100644 index cc6d6b12..00000000 --- a/modules/gcp-workspace-basic/README.md +++ /dev/null @@ -1,67 +0,0 @@ -gcp basic -========================= - -In this template, we show how to deploy a workspace with managed vpc. - - -## Requirements - -- You need to have run gcp-sa-provisionning and have a service account to fill in the variables. -- If you want to deploy to a new project, you will need to grant the custom role generated in that template to the service acount in the new project. -- The Service Account needs to be added as Databricks Admin in the account console - -## Run as an SA - -You can do the same thing by provisionning a service account that will have the same permissions - and associate the key associated to it. - - -## Run the tempalte - -- You need to fill in the variables.tf -- run `terraform init` -- run `teraform apply` - - -## Requirements - -No requirements. - -## Providers - -| Name | Version | -|------|---------| -| [databricks](#provider\_databricks) | n/a | -| [google](#provider\_google) | n/a | -| [random](#provider\_random) | n/a | - -## Modules - -No modules. - -## Resources - -| Name | Type | -|------|------| -| [databricks_mws_workspaces.databricks_workspace](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_workspaces) | resource | -| [random_string.suffix](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/string) | resource | -| [google_client_config.current](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/client_config) | data source | -| [google_client_openid_userinfo.me](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/client_openid_userinfo) | data source | - -## Inputs - -| Name | Description | Type | Default | Required | -|------|-------------|------|---------|:--------:| -| [databricks\_account\_id](#input\_databricks\_account\_id) | Databricks Account ID | `string` | n/a | yes | -| [delegate\_from](#input\_delegate\_from) | Identities to allow to impersonate created service account (in form of user:user.name@example.com, group:deployers@example.com or serviceAccount:sa1@project.iam.gserviceaccount.com) | `list(string)` | n/a | yes | -| [google\_project](#input\_google\_project) | Google project for VCP/workspace deployment | `string` | n/a | yes | -| [google\_region](#input\_google\_region) | Google region for VCP/workspace deployment | `string` | n/a | yes | -| [prefix](#input\_prefix) | Prefix to use in generated VPC name | `string` | n/a | yes | -| [workspace\_name](#input\_workspace\_name) | Name of the workspace to create | `string` | n/a | yes | - -## Outputs - -| Name | Description | -|------|-------------| -| [databricks\_host](#output\_databricks\_host) | n/a | -| [databricks\_token](#output\_databricks\_token) | n/a | - diff --git a/modules/gcp-workspace-basic/init.tf b/modules/gcp-workspace-basic/init.tf deleted file mode 100644 index b07d3474..00000000 --- a/modules/gcp-workspace-basic/init.tf +++ /dev/null @@ -1,24 +0,0 @@ -terraform { - required_providers { - databricks = { - source = "databricks/databricks" - } - google = { - source = "hashicorp/google" - } - } -} - -data "google_client_openid_userinfo" "me" { -} - - -data "google_client_config" "current" { -} - - -resource "random_string" "suffix" { - special = false - upper = false - length = 6 -} diff --git a/modules/gcp-workspace-basic/outputs.tf b/modules/gcp-workspace-basic/outputs.tf deleted file mode 100644 index d6b170a9..00000000 --- a/modules/gcp-workspace-basic/outputs.tf +++ /dev/null @@ -1,9 +0,0 @@ - -output "databricks_host" { - value = databricks_mws_workspaces.databricks_workspace.workspace_url -} - -output "databricks_token" { - value = databricks_mws_workspaces.databricks_workspace.token[0].token_value - sensitive = true -} diff --git a/modules/gcp-workspace-basic/variables.tf b/modules/gcp-workspace-basic/variables.tf deleted file mode 100644 index 5e94a563..00000000 --- a/modules/gcp-workspace-basic/variables.tf +++ /dev/null @@ -1,29 +0,0 @@ -variable "databricks_account_id" { - type = string - description = "Databricks Account ID" -} - -variable "google_project" { - type = string - description = "Google project for VCP/workspace deployment" -} - -variable "google_region" { - type = string - description = "Google region for VCP/workspace deployment" -} - -variable "prefix" { - type = string - description = "Prefix to use in generated VPC name" -} - -variable "workspace_name" { - type = string - description = "Name of the workspace to create" -} - -variable "delegate_from" { - description = "Identities to allow to impersonate created service account (in form of user:user.name@example.com, group:deployers@example.com or serviceAccount:sa1@project.iam.gserviceaccount.com)" - type = list(string) -} diff --git a/modules/gcp-workspace-basic/workspace.tf b/modules/gcp-workspace-basic/workspace.tf deleted file mode 100644 index 262d8a06..00000000 --- a/modules/gcp-workspace-basic/workspace.tf +++ /dev/null @@ -1,14 +0,0 @@ -resource "databricks_mws_workspaces" "databricks_workspace" { - account_id = var.databricks_account_id - workspace_name = var.workspace_name - - location = var.google_region - cloud_resource_container { - gcp { - project_id = var.google_project - } - } - token { - comment = "Terraform token" - } -} diff --git a/modules/gcp-workspace-byovpc/Makefile b/modules/gcp-workspace-byovpc/Makefile deleted file mode 100644 index 653039d8..00000000 --- a/modules/gcp-workspace-byovpc/Makefile +++ /dev/null @@ -1,7 +0,0 @@ -.PHONY: docs test_docs - -docs: - terraform-docs -c ../../.terraform-docs.yml . - -test_docs: - terraform-docs -c ../../.terraform-docs.yml --output-check . diff --git a/modules/gcp-workspace-byovpc/README.md b/modules/gcp-workspace-byovpc/README.md deleted file mode 100644 index 0fbaf403..00000000 --- a/modules/gcp-workspace-byovpc/README.md +++ /dev/null @@ -1,75 +0,0 @@ -gcp byovpc -========================= - -In this template, we show how to deploy a workspace with a custom VPC. - - -## Requirements - -- You need to have run `gcp-sa-provisionning` module and have a service account to fill in the variables. -- If you want to deploy to a new project, you will need to grant the custom role generated in that template to the service acount in the new project. -- The sizing of the custom vpc subnets needs to be appropriate for the usage of the workspace. [This documentation covers it](https://docs.gcp.databricks.com/administration-guide/cloud-configurations/gcp/network-sizing.html) - -## Run as an SA - -You can do the same thing by provisionning a service account that will have the same permissions - and associate the key associated to it. - - -## Run the tempalte - -- You need to fill in the `variables.tf` -- run `terraform init` -- run `teraform apply` - - -## Requirements - -No requirements. - -## Providers - -| Name | Version | -|------|---------| -| [databricks](#provider\_databricks) | n/a | -| [google](#provider\_google) | n/a | -| [random](#provider\_random) | n/a | - -## Modules - -No modules. - -## Resources - -| Name | Type | -|------|------| -| [databricks_mws_networks.databricks_network](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_networks) | resource | -| [databricks_mws_workspaces.databricks_workspace](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_workspaces) | resource | -| [google_compute_network.dbx_private_vpc](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_network) | resource | -| [google_compute_router.router](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_router) | resource | -| [google_compute_router_nat.nat](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_router_nat) | resource | -| [google_compute_subnetwork.network-with-private-secondary-ip-ranges](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_subnetwork) | resource | -| [random_string.suffix](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/string) | resource | -| [google_client_config.current](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/client_config) | data source | -| [google_client_openid_userinfo.me](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/client_openid_userinfo) | data source | - -## Inputs - -| Name | Description | Type | Default | Required | -|------|-------------|------|---------|:--------:| -| [databricks\_account\_id](#input\_databricks\_account\_id) | Databricks Account ID | `string` | n/a | yes | -| [delegate\_from](#input\_delegate\_from) | Identities to allow to impersonate created service account (in form of user:user.name@example.com, group:deployers@example.com or serviceAccount:sa1@project.iam.gserviceaccount.com) | `list(string)` | n/a | yes | -| [google\_project](#input\_google\_project) | Google project for VCP/workspace deployment | `string` | n/a | yes | -| [google\_region](#input\_google\_region) | Google region for VCP/workspace deployment | `string` | n/a | yes | -| [nat\_name](#input\_nat\_name) | Name of the NAT service in compute router | `string` | n/a | yes | -| [prefix](#input\_prefix) | Prefix to use in generated VPC name | `string` | n/a | yes | -| [router\_name](#input\_router\_name) | Name of the compute router to create | `string` | n/a | yes | -| [subnet\_ip\_cidr\_range](#input\_subnet\_ip\_cidr\_range) | IP Range for Nodes subnet (primary) | `string` | n/a | yes | -| [subnet\_name](#input\_subnet\_name) | Name of the subnet to create | `string` | n/a | yes | - -## Outputs - -| Name | Description | -|------|-------------| -| [databricks\_host](#output\_databricks\_host) | n/a | -| [databricks\_token](#output\_databricks\_token) | n/a | - \ No newline at end of file diff --git a/modules/gcp-workspace-byovpc/init.tf b/modules/gcp-workspace-byovpc/init.tf deleted file mode 100644 index 103a33ee..00000000 --- a/modules/gcp-workspace-byovpc/init.tf +++ /dev/null @@ -1,22 +0,0 @@ -terraform { - required_providers { - databricks = { - source = "databricks/databricks" - } - google = { - source = "hashicorp/google" - } - } -} - -data "google_client_openid_userinfo" "me" { -} - -data "google_client_config" "current" { -} - -resource "random_string" "suffix" { - special = false - upper = false - length = 6 -} diff --git a/modules/gcp-workspace-byovpc/outputs.tf b/modules/gcp-workspace-byovpc/outputs.tf deleted file mode 100644 index f544b3ba..00000000 --- a/modules/gcp-workspace-byovpc/outputs.tf +++ /dev/null @@ -1,8 +0,0 @@ -output "databricks_host" { - value = databricks_mws_workspaces.databricks_workspace.workspace_url -} - -output "databricks_token" { - value = databricks_mws_workspaces.databricks_workspace.token[0].token_value - sensitive = true -} \ No newline at end of file diff --git a/modules/gcp-workspace-byovpc/variables.tf b/modules/gcp-workspace-byovpc/variables.tf deleted file mode 100644 index 45650087..00000000 --- a/modules/gcp-workspace-byovpc/variables.tf +++ /dev/null @@ -1,45 +0,0 @@ -variable "databricks_account_id" { - type = string - description = "Databricks Account ID" -} - -variable "google_project" { - type = string - description = "Google project for VCP/workspace deployment" -} - -variable "google_region" { - type = string - description = "Google region for VCP/workspace deployment" -} - -variable "prefix" { - type = string - description = "Prefix to use in generated VPC name" -} - -# These three ranges need to be computed based on the workspace size (cf documentation) -variable "subnet_ip_cidr_range" { - type = string - description = "IP Range for Nodes subnet (primary)" -} - -variable "subnet_name" { - type = string - description = "Name of the subnet to create" -} - -variable "router_name" { - type = string - description = "Name of the compute router to create" -} - -variable "nat_name" { - type = string - description = "Name of the NAT service in compute router" -} - -variable "delegate_from" { - description = "Identities to allow to impersonate created service account (in form of user:user.name@example.com, group:deployers@example.com or serviceAccount:sa1@project.iam.gserviceaccount.com)" - type = list(string) -} diff --git a/modules/gcp-workspace-byovpc/vpc.tf b/modules/gcp-workspace-byovpc/vpc.tf deleted file mode 100644 index 31e8e808..00000000 --- a/modules/gcp-workspace-byovpc/vpc.tf +++ /dev/null @@ -1,40 +0,0 @@ -resource "google_compute_network" "dbx_private_vpc" { - project = var.google_project - name = "${var.prefix}-${random_string.suffix.result}" - auto_create_subnetworks = false -} - -resource "google_compute_subnetwork" "network-with-private-secondary-ip-ranges" { - name = var.subnet_name - ip_cidr_range = var.subnet_ip_cidr_range - region = var.google_region - network = google_compute_network.dbx_private_vpc.id - private_ip_google_access = true -} - -resource "google_compute_router" "router" { - name = var.router_name - region = google_compute_subnetwork.network-with-private-secondary-ip-ranges.region - network = google_compute_network.dbx_private_vpc.id -} - -resource "google_compute_router_nat" "nat" { - name = var.nat_name - router = google_compute_router.router.name - region = google_compute_router.router.region - nat_ip_allocate_option = "AUTO_ONLY" - source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES" -} - -resource "databricks_mws_networks" "databricks_network" { - account_id = var.databricks_account_id - - network_name = "${var.prefix}-${random_string.suffix.result}" - - gcp_network_info { - network_project_id = var.google_project - vpc_id = google_compute_network.dbx_private_vpc.name - subnet_id = google_compute_subnetwork.network-with-private-secondary-ip-ranges.name - subnet_region = google_compute_subnetwork.network-with-private-secondary-ip-ranges.region - } -} diff --git a/modules/gcp-workspace-byovpc/workspace.tf b/modules/gcp-workspace-byovpc/workspace.tf deleted file mode 100644 index 0f7c7a0a..00000000 --- a/modules/gcp-workspace-byovpc/workspace.tf +++ /dev/null @@ -1,20 +0,0 @@ -resource "databricks_mws_workspaces" "databricks_workspace" { - account_id = var.databricks_account_id - workspace_name = "dbx-example-tf-deploy-${random_string.suffix.result}" - - location = var.google_region - cloud_resource_container { - gcp { - project_id = var.google_project - } - } - - network_id = databricks_mws_networks.databricks_network.network_id - - token { - comment = "Terraform token" - } - - # this makes sure that the NAT is created for outbound traffic before creating the workspace - depends_on = [google_compute_router_nat.nat] -} diff --git a/modules/gcp/Makefile b/modules/gcp/Makefile new file mode 100644 index 00000000..30b525d1 --- /dev/null +++ b/modules/gcp/Makefile @@ -0,0 +1,8 @@ +PROJECTS := $(dir $(wildcard */README.md)) + +docs: $(PROJECTS) + +$(PROJECTS): + $(MAKE) -C $@ docs + +.PHONY: $(PROJECTS) docs diff --git a/modules/gcp/databricks-workspace/Makefile b/modules/gcp/databricks-workspace/Makefile new file mode 100644 index 00000000..38b83c2e --- /dev/null +++ b/modules/gcp/databricks-workspace/Makefile @@ -0,0 +1,3 @@ +.PHONY: docs +docs: + terraform-docs -c ../../../.terraform-docs.yml . diff --git a/modules/gcp/databricks-workspace/README.md b/modules/gcp/databricks-workspace/README.md new file mode 100644 index 00000000..e80401bb --- /dev/null +++ b/modules/gcp/databricks-workspace/README.md @@ -0,0 +1,123 @@ +# GCP Databricks Workspace Composer + +This module creates a complete Databricks workspace on Google Cloud Platform with full networking, connectivity, and authentication management. + +Not to be confused with the workspace submodule (../workspace), which this composer calls to register the workspace with the Databricks control plane. + +## Usage + +```hcl +module "workspace" { + source = "github.com/databricks/terraform-databricks-examples//modules/gcp/databricks-workspace" + + prefix = "acme" + databricks_account_id = var.databricks_account_id + google_project = "my-workspace-project" + google_region = "us-central1" + + vpc_source = { spoke = "databricks_managed" } # spoke: databricks_managed | create | existing; hub (optional): create | existing +} +``` + +See `examples/gcp-basic`, `examples/gcp-byovpc`, `examples/gcp-existing-vpc`, and `examples/gcp-with-psc-exfiltration-protection` for the four supported scenarios. + +## Components + +- **network**: VPC creation or integration (databricks_managed, create, or existing) +- **private_connectivity**: Private Service Connect (PSC) with optional frontend/backend +- **workspace**: Databricks MWS resources and workspace +- **dns**: Private DNS zones for restricted egress scenarios + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.5 | +| [databricks](#requirement\_databricks) | >= 1.81.1 | +| [google](#requirement\_google) | >= 6.0 | +| [random](#requirement\_random) | >= 3.0 | + +## Providers + +| Name | Version | +|------|---------| +| [random](#provider\_random) | 3.9.0 | +| [terraform](#provider\_terraform) | n/a | + +## Modules + +| Name | Source | Version | +|------|--------|---------| +| [dns](#module\_dns) | ../dns | n/a | +| [network](#module\_network) | ../network | n/a | +| [private\_connectivity](#module\_private\_connectivity) | ../private-connectivity | n/a | +| [workspace](#module\_workspace) | ../workspace | n/a | + +## Resources + +| Name | Type | +|------|------| +| [random_string.suffix](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/string) | resource | +| [terraform_data.preconditions](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [databricks\_account\_id](#input\_databricks\_account\_id) | Databricks account ID (GUID) where this workspace will be registered | `string` | n/a | yes | +| [google\_project](#input\_google\_project) | GCP project ID hosting the workspace data plane | `string` | n/a | yes | +| [google\_region](#input\_google\_region) | GCP region where the workspace will be deployed. When any private\_link\_* flag or restricted\_egress is true, the region must be supported by Databricks PSC (see preconditions.tf) | `string` | n/a | yes | +| [prefix](#input\_prefix) | Prefix used to name generated resources (e.g. "acme" produces "acme-spoke-vpc-") | `string` | n/a | yes | +| [cmek\_managed\_services\_key\_id](#input\_cmek\_managed\_services\_key\_id) | Cloud KMS key resource ID for managed-services CMEK (control-plane data: notebooks, secrets, queries). Null disables. The principal running Terraform needs cloudkms.cryptoKeys.getIamPolicy and setIamPolicy on the key - Databricks sets the key's IAM policy at workspace creation. Enterprise tier; set at creation only. The key must exist before plan (a key created in the same configuration makes the count unknown and fails plan) | `string` | `null` | no | +| [cmek\_storage\_key\_id](#input\_cmek\_storage\_key\_id) | Cloud KMS key resource ID for workspace-storage CMEK (GCS buckets and GCE persistent disks). Null disables. Same permission and tier requirements as cmek\_managed\_services\_key\_id; set at creation only. The key must exist before plan (a key created in the same configuration makes the count unknown and fails plan) | `string` | `null` | no | +| [enable\_hub\_spoke\_peering](#input\_enable\_hub\_spoke\_peering) | Create the bidirectional VPC peering between hub and spoke. Disable when hub-spoke connectivity is provided by other means (e.g. Shared VPC or an existing transit). Cloud DNS peering zones do not depend on it. Only takes effect when the hub is enabled | `bool` | `true` | no | +| [existing\_hub\_subnet\_name](#input\_existing\_hub\_subnet\_name) | Name of the pre-existing hub subnet (must be in google\_region). Required when vpc\_source.hub=existing | `string` | `null` | no | +| [existing\_hub\_vpc\_name](#input\_existing\_hub\_vpc\_name) | Name of the pre-existing hub VPC. Required when vpc\_source.hub=existing | `string` | `null` | no | +| [existing\_subnet\_name](#input\_existing\_subnet\_name) | Name of the pre-existing subnet to use (must be in google\_region). Required when vpc\_source.spoke=existing | `string` | `null` | no | +| [existing\_vpc\_name](#input\_existing\_vpc\_name) | Name of the pre-existing VPC to use. Required when vpc\_source.spoke=existing | `string` | `null` | no | +| [hive\_metastore\_ip](#input\_hive\_metastore\_ip) | Regional legacy Hive metastore IP. When set, an egress allow rule (tcp/3306) is created under restricted egress; when null, no rule is created. Workspaces using Unity Catalog (the default) do not need this. Regional IPs: https://docs.databricks.com/gcp/en/resources/ip-domain-region | `string` | `null` | no | +| [hub\_vpc\_cidr](#input\_hub\_vpc\_cidr) | CIDR of the hub subnet (e.g. 10.1.0.0/24). Required when restricted\_egress=true and vpc\_source.hub=create | `string` | `null` | no | +| [hub\_vpc\_google\_project](#input\_hub\_vpc\_google\_project) | GCP project hosting the hub VPC. Required when restricted\_egress=true | `string` | `null` | no | +| [is\_spoke\_vpc\_shared](#input\_is\_spoke\_vpc\_shared) | If true and the spoke VPC project differs from the workspace project, bind the spoke project as a Shared-VPC host and the workspace project as a service project. Works with or without restricted\_egress | `bool` | `false` | no | +| [private\_access\_only](#input\_private\_access\_only) | Create databricks\_mws\_private\_access\_settings with public\_access\_enabled=false. Workspace becomes reachable only through PSC endpoints | `bool` | `false` | no | +| [private\_link\_backend](#input\_private\_link\_backend) | Create the backend (SCC, data plane) PSC endpoint and a backend databricks\_mws\_vpc\_endpoint. On GCP both flags must be enabled together (see preconditions.tf) | `bool` | `false` | no | +| [private\_link\_frontend](#input\_private\_link\_frontend) | Create the frontend (workspace UI/API) PSC endpoint and a frontend databricks\_mws\_vpc\_endpoint. On GCP both flags must be enabled together (see preconditions.tf) | `bool` | `false` | no | +| [psc\_subnet\_cidr](#input\_psc\_subnet\_cidr) | CIDR of the dedicated PSC subnet in the spoke VPC (e.g. 10.0.255.0/28). Required when restricted\_egress=true or any private\_link\_* flag is true | `string` | `null` | no | +| [restricted\_egress](#input\_restricted\_egress) | Create hub VPC + bidirectional peering + deny-egress firewall + private DNS zones. Requires vpc\_source.spoke=create and at least one private\_link\_* flag | `bool` | `false` | no | +| [serverless\_allowed\_internet\_destinations](#input\_serverless\_allowed\_internet\_destinations) | FQDNs serverless workloads may reach when serverless\_egress\_mode=restricted (max 100) | `list(string)` | `[]` | no | +| [serverless\_allowed\_storage\_destinations](#input\_serverless\_allowed\_storage\_destinations) | GCS bucket names serverless workloads may reach when serverless\_egress\_mode=restricted (max 100); region is taken from google\_region | `list(string)` | `[]` | no | +| [serverless\_egress\_enforcement](#input\_serverless\_egress\_enforcement) | enforced: violations are blocked; dry\_run: violations are only logged (use to evaluate a policy before enforcing) | `string` | `"enforced"` | no | +| [serverless\_egress\_mode](#input\_serverless\_egress\_mode) | Serverless egress control. unmanaged: no network policy resources; full: policy with FULL\_ACCESS; restricted: deny-by-default policy allowing only the listed destinations. Requires the workspace to be on the Enterprise tier | `string` | `"unmanaged"` | no | +| [spoke\_vpc\_cidr](#input\_spoke\_vpc\_cidr) | CIDR of the spoke VPC address space (e.g. 10.0.0.0/16). Required when vpc\_source.spoke=create; ignored otherwise | `string` | `null` | no | +| [spoke\_vpc\_google\_project](#input\_spoke\_vpc\_google\_project) | GCP project hosting the spoke VPC. Defaults to google\_project when null | `string` | `null` | no | +| [subnet\_cidr](#input\_subnet\_cidr) | CIDR of the spoke subnet primary range (e.g. 10.0.0.0/22). Required when vpc\_source.spoke=create | `string` | `null` | no | +| [tags](#input\_tags) | Map of tags. Currently not propagated to child resources; reserved for future use | `map(string)` | `{}` | no | +| [vpc\_source](#input\_vpc\_source) | Where the workspace networks come from. spoke: databricks\_managed (no networking module called), create (Terraform creates VPC + subnet + NAT), existing (data-source lookup of existing\_vpc\_name/existing\_subnet\_name). hub — only relevant when spoke is create or existing, and only consumed when restricted\_egress=true: create (hub VPC + subnet created; hub\_vpc\_cidr required; the default when unset) or existing (lookup of existing\_hub\_vpc\_name/existing\_hub\_subnet\_name) | 
object({
    spoke = optional(string, "databricks_managed")
    hub   = optional(string)
  })
| `{}` | no | +| [workspace\_name](#input\_workspace\_name) | Optional workspace name override. Defaults to "prefix-ws-suffix" when null | `string` | `null` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| [backend\_endpoint\_id](#output\_backend\_endpoint\_id) | Backend (SCC) mws\_vpc\_endpoint ID (null when private\_link\_backend=false) | +| [backend\_psc\_ip\_spoke](#output\_backend\_psc\_ip\_spoke) | IP address of the spoke-side backend PSC endpoint (null when no PSC) | +| [frontend\_endpoint\_id](#output\_frontend\_endpoint\_id) | Frontend mws\_vpc\_endpoint ID (null when private\_link\_frontend=false) | +| [frontend\_psc\_ip\_hub](#output\_frontend\_psc\_ip\_hub) | IP address of the hub-side frontend PSC endpoint (null when restricted\_egress=false) | +| [frontend\_psc\_ip\_spoke](#output\_frontend\_psc\_ip\_spoke) | IP address of the spoke-side frontend PSC endpoint (null when no PSC) | +| [google\_region](#output\_google\_region) | Region the workspace was deployed to (echo of input; convenient for downstream modules) | +| [hub\_vpc\_id](#output\_hub\_vpc\_id) | Hub VPC ID (null when restricted\_egress=false) | +| [hub\_vpc\_self\_link](#output\_hub\_vpc\_self\_link) | Hub VPC self-link (null when restricted\_egress=false) | +| [nat\_id](#output\_nat\_id) | Cloud NAT ID (null when vpc\_source.spoke != create or when restricted\_egress=true) | +| [network\_id](#output\_network\_id) | databricks\_mws\_networks ID (null when vpc\_source.spoke=databricks\_managed) | +| [private\_access\_settings\_id](#output\_private\_access\_settings\_id) | databricks\_mws\_private\_access\_settings ID (null when private\_access\_only=false) | +| [serverless\_network\_policy\_id](#output\_serverless\_network\_policy\_id) | Serverless egress network-policy ID bound to the workspace (null when serverless\_egress\_mode=unmanaged) | +| [spoke\_subnet\_id](#output\_spoke\_subnet\_id) | Spoke subnet ID (null when vpc\_source.spoke=databricks\_managed) | +| [spoke\_subnet\_self\_link](#output\_spoke\_subnet\_self\_link) | Spoke subnet self-link (null when vpc\_source.spoke=databricks\_managed) | +| [spoke\_vpc\_id](#output\_spoke\_vpc\_id) | Spoke VPC ID (null when vpc\_source.spoke=databricks\_managed) | +| [spoke\_vpc\_self\_link](#output\_spoke\_vpc\_self\_link) | Spoke VPC self-link (null when vpc\_source.spoke=databricks\_managed) | +| [suffix](#output\_suffix) | Random suffix used in resource names (useful when wiring downstream modules) | +| [transit\_endpoint\_id](#output\_transit\_endpoint\_id) | Hub-side mws\_vpc\_endpoint ID (null when no hub or no frontend PSC) | +| [workspace\_id](#output\_workspace\_id) | Databricks workspace ID | +| [workspace\_url](#output\_workspace\_url) | Databricks workspace URL (https://..gcp.databricks.com) | + \ No newline at end of file diff --git a/modules/gcp/databricks-workspace/locals.tf b/modules/gcp/databricks-workspace/locals.tf new file mode 100644 index 00000000..af678f19 --- /dev/null +++ b/modules/gcp/databricks-workspace/locals.tf @@ -0,0 +1,18 @@ +locals { + databricks_managed = var.vpc_source.spoke == "databricks_managed" + create_spoke = var.vpc_source.spoke == "create" + use_existing_spoke = var.vpc_source.spoke == "existing" + hub_source = var.vpc_source.hub != null ? var.vpc_source.hub : "create" + + any_private_link = var.private_link_frontend || var.private_link_backend + spoke_project = coalesce(var.spoke_vpc_google_project, var.google_project) + + # Both submodules below consume module.network[0] outputs (spoke/hub VPC refs), which + # don't exist when vpc_source.spoke="databricks_managed". Valid configs never combine + # databricks_managed with any_private_link/restricted_egress (see preconditions.tf), but + # negative fixtures deliberately do; gating on !databricks_managed here (in addition to the + # precondition) keeps plan from cascading into "Invalid index" / "Missing required argument" + # errors before the precondition message is shown. + private_connectivity_enabled = local.any_private_link && !local.databricks_managed + dns_enabled = var.restricted_egress && !local.databricks_managed +} diff --git a/modules/gcp/databricks-workspace/main.tf b/modules/gcp/databricks-workspace/main.tf new file mode 100644 index 00000000..bd1c859c --- /dev/null +++ b/modules/gcp/databricks-workspace/main.tf @@ -0,0 +1,109 @@ +module "network" { + source = "../network" + count = local.databricks_managed ? 0 : 1 + + prefix = var.prefix + suffix = random_string.suffix.result + google_region = var.google_region + vpc_source = var.vpc_source.spoke + spoke_vpc_google_project = local.spoke_project + + subnet_cidr = var.subnet_cidr + + existing_vpc_name = var.existing_vpc_name + existing_subnet_name = var.existing_subnet_name + + enable_hub = var.restricted_egress + enable_hub_spoke_peering = var.enable_hub_spoke_peering + hub_vpc_source = local.hub_source + existing_hub_vpc_name = var.existing_hub_vpc_name + existing_hub_subnet_name = var.existing_hub_subnet_name + hub_vpc_google_project = var.hub_vpc_google_project + hub_vpc_cidr = var.hub_vpc_cidr + is_spoke_vpc_shared = var.is_spoke_vpc_shared + workspace_google_project = var.google_project + enable_nat = !var.restricted_egress +} + +module "private_connectivity" { + source = "../private-connectivity" + count = local.private_connectivity_enabled ? 1 : 0 + + prefix = var.prefix + suffix = random_string.suffix.result + google_region = var.google_region + + spoke_vpc_id = local.databricks_managed ? null : module.network[0].spoke_vpc_id + spoke_vpc_self_link = local.databricks_managed ? null : module.network[0].spoke_vpc_self_link + spoke_vpc_google_project = local.spoke_project + spoke_vpc_cidr = var.spoke_vpc_cidr + + hub_vpc_id = var.restricted_egress ? module.network[0].hub_vpc_id : null + hub_vpc_self_link = var.restricted_egress ? module.network[0].hub_vpc_self_link : null + hub_vpc_google_project = var.hub_vpc_google_project + hub_subnet_name = var.restricted_egress ? module.network[0].hub_subnet_name : null + + enable_frontend = var.private_link_frontend + enable_backend = var.private_link_backend + restrict_egress = var.restricted_egress + enable_hub = var.restricted_egress + psc_subnet_cidr = var.psc_subnet_cidr + + hive_metastore_ip = var.hive_metastore_ip +} + +module "workspace" { + source = "../workspace" + + prefix = var.prefix + suffix = random_string.suffix.result + workspace_name = var.workspace_name + databricks_account_id = var.databricks_account_id + google_project = var.google_project + google_region = var.google_region + vpc_source = var.vpc_source.spoke + + spoke_vpc_name = local.databricks_managed ? null : module.network[0].spoke_vpc_name + spoke_subnet_name = local.databricks_managed ? null : module.network[0].spoke_subnet_name + spoke_vpc_google_project = local.spoke_project + hub_vpc_google_project = var.hub_vpc_google_project + + frontend_forwarding_rule_name = local.private_connectivity_enabled ? module.private_connectivity[0].frontend_forwarding_rule_name : null + backend_forwarding_rule_name = local.private_connectivity_enabled ? module.private_connectivity[0].backend_forwarding_rule_name : null + hub_frontend_forwarding_rule_name = local.private_connectivity_enabled ? module.private_connectivity[0].hub_frontend_forwarding_rule_name : null + + enable_frontend = var.private_link_frontend + enable_backend = var.private_link_backend + private_access_only = var.private_access_only + enable_hub = var.restricted_egress + + nat_dependency = local.databricks_managed ? null : module.network[0].nat_id + + serverless_egress_mode = var.serverless_egress_mode + serverless_allowed_internet_destinations = var.serverless_allowed_internet_destinations + serverless_allowed_storage_destinations = var.serverless_allowed_storage_destinations + serverless_egress_enforcement = var.serverless_egress_enforcement + + cmek_managed_services_key_id = var.cmek_managed_services_key_id + cmek_storage_key_id = var.cmek_storage_key_id +} + +module "dns" { + source = "../dns" + count = local.dns_enabled ? 1 : 0 + + prefix = var.prefix + google_region = var.google_region + + hub_vpc_id = local.databricks_managed ? null : module.network[0].hub_vpc_id + hub_vpc_google_project = var.hub_vpc_google_project + + spoke_vpc_id = local.databricks_managed ? null : module.network[0].spoke_vpc_id + spoke_vpc_google_project = local.spoke_project + + workspace_url = module.workspace.workspace_url + + frontend_psc_ip_spoke = local.private_connectivity_enabled ? module.private_connectivity[0].frontend_psc_ip_spoke : null + frontend_psc_ip_hub = local.private_connectivity_enabled ? module.private_connectivity[0].frontend_psc_ip_hub : null + backend_psc_ip_spoke = local.private_connectivity_enabled ? module.private_connectivity[0].backend_psc_ip_spoke : null +} diff --git a/modules/gcp/databricks-workspace/outputs.tf b/modules/gcp/databricks-workspace/outputs.tf new file mode 100644 index 00000000..78049550 --- /dev/null +++ b/modules/gcp/databricks-workspace/outputs.tf @@ -0,0 +1,104 @@ +# === Workspace =========================================================== +output "workspace_id" { + value = module.workspace.workspace_id + description = "Databricks workspace ID" +} + +output "workspace_url" { + value = module.workspace.workspace_url + description = "Databricks workspace URL (https://..gcp.databricks.com)" +} + +output "network_id" { + value = module.workspace.network_id + description = "databricks_mws_networks ID (null when vpc_source.spoke=databricks_managed)" +} + +output "private_access_settings_id" { + value = module.workspace.private_access_settings_id + description = "databricks_mws_private_access_settings ID (null when private_access_only=false)" +} + +output "serverless_network_policy_id" { + value = module.workspace.serverless_network_policy_id + description = "Serverless egress network-policy ID bound to the workspace (null when serverless_egress_mode=unmanaged)" +} + +# === mws_vpc_endpoint IDs (Databricks-side PSC registration) ============ +output "frontend_endpoint_id" { + value = module.workspace.frontend_endpoint_id + description = "Frontend mws_vpc_endpoint ID (null when private_link_frontend=false)" +} + +output "backend_endpoint_id" { + value = module.workspace.backend_endpoint_id + description = "Backend (SCC) mws_vpc_endpoint ID (null when private_link_backend=false)" +} + +output "transit_endpoint_id" { + value = module.workspace.transit_endpoint_id + description = "Hub-side mws_vpc_endpoint ID (null when no hub or no frontend PSC)" +} + +# === Network ============================================================= +output "spoke_vpc_id" { + value = local.databricks_managed ? null : module.network[0].spoke_vpc_id + description = "Spoke VPC ID (null when vpc_source.spoke=databricks_managed)" +} + +output "spoke_vpc_self_link" { + value = local.databricks_managed ? null : module.network[0].spoke_vpc_self_link + description = "Spoke VPC self-link (null when vpc_source.spoke=databricks_managed)" +} + +output "spoke_subnet_id" { + value = local.databricks_managed ? null : module.network[0].spoke_subnet_id + description = "Spoke subnet ID (null when vpc_source.spoke=databricks_managed)" +} + +output "spoke_subnet_self_link" { + value = local.databricks_managed ? null : module.network[0].spoke_subnet_self_link + description = "Spoke subnet self-link (null when vpc_source.spoke=databricks_managed)" +} + +output "hub_vpc_id" { + value = local.dns_enabled ? module.network[0].hub_vpc_id : null + description = "Hub VPC ID (null when restricted_egress=false)" +} + +output "hub_vpc_self_link" { + value = local.dns_enabled ? module.network[0].hub_vpc_self_link : null + description = "Hub VPC self-link (null when restricted_egress=false)" +} + +output "nat_id" { + value = local.create_spoke && !var.restricted_egress ? module.network[0].nat_id : null + description = "Cloud NAT ID (null when vpc_source.spoke != create or when restricted_egress=true)" +} + +# === Private connectivity =============================================== +output "frontend_psc_ip_spoke" { + value = local.private_connectivity_enabled ? module.private_connectivity[0].frontend_psc_ip_spoke : null + description = "IP address of the spoke-side frontend PSC endpoint (null when no PSC)" +} + +output "backend_psc_ip_spoke" { + value = local.private_connectivity_enabled ? module.private_connectivity[0].backend_psc_ip_spoke : null + description = "IP address of the spoke-side backend PSC endpoint (null when no PSC)" +} + +output "frontend_psc_ip_hub" { + value = local.private_connectivity_enabled ? module.private_connectivity[0].frontend_psc_ip_hub : null + description = "IP address of the hub-side frontend PSC endpoint (null when restricted_egress=false)" +} + +# === Identifiers ======================================================== +output "suffix" { + value = random_string.suffix.result + description = "Random suffix used in resource names (useful when wiring downstream modules)" +} + +output "google_region" { + value = var.google_region + description = "Region the workspace was deployed to (echo of input; convenient for downstream modules)" +} diff --git a/modules/gcp/databricks-workspace/preconditions.tf b/modules/gcp/databricks-workspace/preconditions.tf new file mode 100644 index 00000000..a198914e --- /dev/null +++ b/modules/gcp/databricks-workspace/preconditions.tf @@ -0,0 +1,59 @@ +# Cross-variable preconditions. +resource "terraform_data" "preconditions" { + lifecycle { + precondition { + condition = !var.restricted_egress || local.create_spoke + error_message = "restricted_egress=true requires vpc_source.spoke=\"create\" (hub-spoke topology needs us to own the spoke VPC)." + } + precondition { + condition = !var.restricted_egress || local.any_private_link + error_message = "restricted_egress=true requires at least one of private_link_frontend or private_link_backend." + } + precondition { + condition = var.private_link_frontend == var.private_link_backend + error_message = "On GCP, private_link_frontend and private_link_backend must be enabled together: databricks_mws_networks.vpc_endpoints requires both dataplane_relay and rest_api endpoint references. (The flags stay independent in the cross-cloud contract for clouds that support single-sided PrivateLink.)" + } + precondition { + condition = !local.any_private_link || var.psc_subnet_cidr != null + error_message = "psc_subnet_cidr is required when any private_link_* flag is true." + } + precondition { + condition = !var.restricted_egress || var.hub_vpc_google_project != null + error_message = "restricted_egress=true requires hub_vpc_google_project." + } + precondition { + condition = !(var.restricted_egress && local.hub_source == "create") || var.hub_vpc_cidr != null + error_message = "restricted_egress=true with vpc_source.hub=\"create\" requires hub_vpc_cidr." + } + precondition { + condition = !(var.restricted_egress && local.hub_source == "existing") || (var.existing_hub_vpc_name != null && var.existing_hub_subnet_name != null) + error_message = "vpc_source.hub=\"existing\" requires existing_hub_vpc_name and existing_hub_subnet_name." + } + precondition { + condition = !local.create_spoke || (var.spoke_vpc_cidr != null && var.subnet_cidr != null) + error_message = "vpc_source.spoke=\"create\" requires spoke_vpc_cidr and subnet_cidr." + } + precondition { + condition = !local.use_existing_spoke || (var.existing_vpc_name != null && var.existing_subnet_name != null) + error_message = "vpc_source.spoke=\"existing\" requires existing_vpc_name and existing_subnet_name." + } + precondition { + condition = !local.databricks_managed || (!var.private_link_frontend && !var.private_link_backend && !var.restricted_egress) + error_message = "vpc_source.spoke=\"databricks_managed\" forbids private_link_frontend, private_link_backend, and restricted_egress." + } + precondition { + condition = ( + !local.any_private_link && !var.restricted_egress + ) || contains([ + "asia-northeast1", "asia-south1", "asia-southeast1", "australia-southeast1", + "europe-west1", "europe-west2", "europe-west3", "northamerica-northeast1", + "southamerica-east1", "us-central1", "us-east1", "us-east4", "us-west1", "us-west4" + ], var.google_region) + error_message = "google_region must be a region supported by Databricks PSC when any private_link_* flag or restricted_egress is true." + } + precondition { + condition = var.serverless_egress_mode == "restricted" || (length(var.serverless_allowed_internet_destinations) == 0 && length(var.serverless_allowed_storage_destinations) == 0) + error_message = "serverless_allowed_internet_destinations and serverless_allowed_storage_destinations require serverless_egress_mode=\"restricted\"." + } + } +} diff --git a/modules/gcp/databricks-workspace/random.tf b/modules/gcp/databricks-workspace/random.tf new file mode 100644 index 00000000..7c4efc8d --- /dev/null +++ b/modules/gcp/databricks-workspace/random.tf @@ -0,0 +1,9 @@ +resource "random_string" "suffix" { + length = 6 + special = false + upper = false + + lifecycle { + ignore_changes = [special, upper] + } +} diff --git a/modules/gcp/databricks-workspace/tests/basic/main.tf b/modules/gcp/databricks-workspace/tests/basic/main.tf new file mode 100644 index 00000000..9a3e335d --- /dev/null +++ b/modules/gcp/databricks-workspace/tests/basic/main.tf @@ -0,0 +1,28 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { source = "databricks/databricks" } + google = { source = "hashicorp/google" } + } +} + +provider "google" { + project = "fixture-workspace" + region = "us-central1" +} + +provider "databricks" { + host = "https://accounts.gcp.databricks.com" + account_id = "00000000-0000-0000-0000-000000000000" +} + +module "workspace" { + source = "../.." + + prefix = "fixture" + databricks_account_id = "00000000-0000-0000-0000-000000000000" + google_project = "fixture-workspace" + google_region = "us-central1" + + vpc_source = { spoke = "databricks_managed" } +} diff --git a/modules/gcp/databricks-workspace/tests/byovpc/main.tf b/modules/gcp/databricks-workspace/tests/byovpc/main.tf new file mode 100644 index 00000000..c9599751 --- /dev/null +++ b/modules/gcp/databricks-workspace/tests/byovpc/main.tf @@ -0,0 +1,30 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { source = "databricks/databricks" } + google = { source = "hashicorp/google" } + } +} + +provider "google" { + project = "fixture-workspace" + region = "us-central1" +} + +provider "databricks" { + host = "https://accounts.gcp.databricks.com" + account_id = "00000000-0000-0000-0000-000000000000" +} + +module "workspace" { + source = "../.." + + prefix = "fixture" + databricks_account_id = "00000000-0000-0000-0000-000000000000" + google_project = "fixture-workspace" + google_region = "us-central1" + + vpc_source = { spoke = "create" } + spoke_vpc_cidr = "10.0.0.0/16" + subnet_cidr = "10.0.0.0/22" +} diff --git a/modules/gcp/databricks-workspace/tests/existing-vpc/main.tf b/modules/gcp/databricks-workspace/tests/existing-vpc/main.tf new file mode 100644 index 00000000..23991340 --- /dev/null +++ b/modules/gcp/databricks-workspace/tests/existing-vpc/main.tf @@ -0,0 +1,30 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { source = "databricks/databricks" } + google = { source = "hashicorp/google" } + } +} + +provider "google" { + project = "fixture-workspace" + region = "us-central1" +} + +provider "databricks" { + host = "https://accounts.gcp.databricks.com" + account_id = "00000000-0000-0000-0000-000000000000" +} + +module "workspace" { + source = "../.." + + prefix = "fixture" + databricks_account_id = "00000000-0000-0000-0000-000000000000" + google_project = "fixture-workspace" + google_region = "us-central1" + + vpc_source = { spoke = "existing" } + existing_vpc_name = "preexisting-vpc" + existing_subnet_name = "preexisting-subnet" +} diff --git a/modules/gcp/databricks-workspace/tests/negative-existing-hub-missing-names/main.tf b/modules/gcp/databricks-workspace/tests/negative-existing-hub-missing-names/main.tf new file mode 100644 index 00000000..d4b57e90 --- /dev/null +++ b/modules/gcp/databricks-workspace/tests/negative-existing-hub-missing-names/main.tf @@ -0,0 +1,47 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { source = "databricks/databricks" } + google = { source = "hashicorp/google" } + } +} + +provider "google" { + project = "fixture-workspace" + region = "us-central1" +} + +provider "databricks" { + host = "https://accounts.gcp.databricks.com" + account_id = "00000000-0000-0000-0000-000000000000" +} + +# precondition fail: vpc_source.hub="existing" requires existing_hub_vpc_name and existing_hub_subnet_name +module "workspace" { + source = "../.." + + prefix = "fixture" + databricks_account_id = "00000000-0000-0000-0000-000000000000" + google_project = "fixture-workspace" + google_region = "us-central1" + + vpc_source = { spoke = "create", hub = "existing" } + spoke_vpc_cidr = "10.0.0.0/16" + subnet_cidr = "10.0.0.0/22" + + private_link_frontend = true + private_link_backend = true + private_access_only = true + restricted_egress = true + + spoke_vpc_google_project = "fixture-spoke" + hub_vpc_google_project = "fixture-hub" + is_spoke_vpc_shared = true + psc_subnet_cidr = "10.0.255.0/28" + + serverless_egress_mode = "restricted" + serverless_allowed_internet_destinations = ["pypi.org"] + + cmek_managed_services_key_id = "projects/fixture-workspace/locations/us-central1/keyRings/fixture-kr/cryptoKeys/fixture-ms-key" + cmek_storage_key_id = "projects/fixture-workspace/locations/us-central1/keyRings/fixture-kr/cryptoKeys/fixture-storage-key" +} diff --git a/modules/gcp/databricks-workspace/tests/negative-existing-missing-name/main.tf b/modules/gcp/databricks-workspace/tests/negative-existing-missing-name/main.tf new file mode 100644 index 00000000..bf232bae --- /dev/null +++ b/modules/gcp/databricks-workspace/tests/negative-existing-missing-name/main.tf @@ -0,0 +1,29 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { source = "databricks/databricks" } + google = { source = "hashicorp/google" } + } +} + +provider "google" { + project = "fixture-workspace" + region = "us-central1" +} + +provider "databricks" { + host = "https://accounts.gcp.databricks.com" + account_id = "00000000-0000-0000-0000-000000000000" +} + +# precondition fail: vpc_source.spoke="existing" requires existing_vpc_name + existing_subnet_name +module "workspace" { + source = "../.." + + prefix = "fixture" + databricks_account_id = "00000000-0000-0000-0000-000000000000" + google_project = "fixture-workspace" + google_region = "us-central1" + + vpc_source = { spoke = "existing" } +} diff --git a/modules/gcp/databricks-workspace/tests/negative-managed-with-hub/main.tf b/modules/gcp/databricks-workspace/tests/negative-managed-with-hub/main.tf new file mode 100644 index 00000000..822c9014 --- /dev/null +++ b/modules/gcp/databricks-workspace/tests/negative-managed-with-hub/main.tf @@ -0,0 +1,29 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { source = "databricks/databricks" } + google = { source = "hashicorp/google" } + } +} + +provider "google" { + project = "fixture-workspace" + region = "us-central1" +} + +provider "databricks" { + host = "https://accounts.gcp.databricks.com" + account_id = "00000000-0000-0000-0000-000000000000" +} + +# validation fail: vpc_source.hub is only relevant when vpc_source.spoke is create or existing +module "workspace" { + source = "../.." + + prefix = "fixture" + databricks_account_id = "00000000-0000-0000-0000-000000000000" + google_project = "fixture-workspace" + google_region = "us-central1" + + vpc_source = { spoke = "databricks_managed", hub = "create" } +} diff --git a/modules/gcp/databricks-workspace/tests/negative-managed-with-psc/main.tf b/modules/gcp/databricks-workspace/tests/negative-managed-with-psc/main.tf new file mode 100644 index 00000000..af04c506 --- /dev/null +++ b/modules/gcp/databricks-workspace/tests/negative-managed-with-psc/main.tf @@ -0,0 +1,30 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { source = "databricks/databricks" } + google = { source = "hashicorp/google" } + } +} + +provider "google" { + project = "fixture-workspace" + region = "us-central1" +} + +provider "databricks" { + host = "https://accounts.gcp.databricks.com" + account_id = "00000000-0000-0000-0000-000000000000" +} + +# precondition fail: vpc_source.spoke="databricks_managed" forbids private_link_frontend +module "workspace" { + source = "../.." + + prefix = "fixture" + databricks_account_id = "00000000-0000-0000-0000-000000000000" + google_project = "fixture-workspace" + google_region = "us-central1" + + vpc_source = { spoke = "databricks_managed" } + private_link_frontend = true +} diff --git a/modules/gcp/databricks-workspace/tests/negative-psc-missing-subnet-cidr/main.tf b/modules/gcp/databricks-workspace/tests/negative-psc-missing-subnet-cidr/main.tf new file mode 100644 index 00000000..0413ca96 --- /dev/null +++ b/modules/gcp/databricks-workspace/tests/negative-psc-missing-subnet-cidr/main.tf @@ -0,0 +1,34 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { source = "databricks/databricks" } + google = { source = "hashicorp/google" } + } +} + +provider "google" { + project = "fixture-workspace" + region = "us-central1" +} + +provider "databricks" { + host = "https://accounts.gcp.databricks.com" + account_id = "00000000-0000-0000-0000-000000000000" +} + +# precondition fail: PSC flags without psc_subnet_cidr +module "workspace" { + source = "../.." + + prefix = "fixture" + databricks_account_id = "00000000-0000-0000-0000-000000000000" + google_project = "fixture-workspace" + google_region = "us-central1" + + vpc_source = { spoke = "create" } + spoke_vpc_cidr = "10.0.0.0/16" + subnet_cidr = "10.0.0.0/22" + + private_link_frontend = true + private_link_backend = true +} diff --git a/modules/gcp/databricks-workspace/tests/negative-psc-single-sided/main.tf b/modules/gcp/databricks-workspace/tests/negative-psc-single-sided/main.tf new file mode 100644 index 00000000..314ead72 --- /dev/null +++ b/modules/gcp/databricks-workspace/tests/negative-psc-single-sided/main.tf @@ -0,0 +1,35 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { source = "databricks/databricks" } + google = { source = "hashicorp/google" } + } +} + +provider "google" { + project = "fixture-workspace" + region = "us-central1" +} + +provider "databricks" { + host = "https://accounts.gcp.databricks.com" + account_id = "00000000-0000-0000-0000-000000000000" +} + +# precondition fail: frontend without backend (GCP requires both) +module "workspace" { + source = "../.." + + prefix = "fixture" + databricks_account_id = "00000000-0000-0000-0000-000000000000" + google_project = "fixture-workspace" + google_region = "us-central1" + + vpc_source = { spoke = "create" } + spoke_vpc_cidr = "10.0.0.0/16" + subnet_cidr = "10.0.0.0/22" + + private_link_frontend = true + private_link_backend = false + psc_subnet_cidr = "10.0.255.0/28" +} diff --git a/modules/gcp/databricks-workspace/tests/negative-restricted-egress-managed/main.tf b/modules/gcp/databricks-workspace/tests/negative-restricted-egress-managed/main.tf new file mode 100644 index 00000000..086f1cd6 --- /dev/null +++ b/modules/gcp/databricks-workspace/tests/negative-restricted-egress-managed/main.tf @@ -0,0 +1,30 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { source = "databricks/databricks" } + google = { source = "hashicorp/google" } + } +} + +provider "google" { + project = "fixture-workspace" + region = "us-central1" +} + +provider "databricks" { + host = "https://accounts.gcp.databricks.com" + account_id = "00000000-0000-0000-0000-000000000000" +} + +# precondition fail: restricted_egress=true requires vpc_source.spoke="create" +module "workspace" { + source = "../.." + + prefix = "fixture" + databricks_account_id = "00000000-0000-0000-0000-000000000000" + google_project = "fixture-workspace" + google_region = "us-central1" + + vpc_source = { spoke = "databricks_managed" } + restricted_egress = true +} diff --git a/modules/gcp/databricks-workspace/tests/negative-restricted-egress-missing-hub/main.tf b/modules/gcp/databricks-workspace/tests/negative-restricted-egress-missing-hub/main.tf new file mode 100644 index 00000000..f91aa547 --- /dev/null +++ b/modules/gcp/databricks-workspace/tests/negative-restricted-egress-missing-hub/main.tf @@ -0,0 +1,34 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { source = "databricks/databricks" } + google = { source = "hashicorp/google" } + } +} + +provider "google" { + project = "fixture-workspace" + region = "us-central1" +} + +provider "databricks" { + host = "https://accounts.gcp.databricks.com" + account_id = "00000000-0000-0000-0000-000000000000" +} + +# precondition fail: restricted_egress=true requires hub_vpc_google_project +module "workspace" { + source = "../.." + + prefix = "fixture" + databricks_account_id = "00000000-0000-0000-0000-000000000000" + google_project = "fixture-workspace" + google_region = "us-central1" + + vpc_source = { spoke = "create" } + spoke_vpc_cidr = "10.0.0.0/16" + subnet_cidr = "10.0.0.0/22" + private_link_frontend = true + private_link_backend = true + restricted_egress = true +} diff --git a/modules/gcp/databricks-workspace/tests/negative-serverless-destinations/main.tf b/modules/gcp/databricks-workspace/tests/negative-serverless-destinations/main.tf new file mode 100644 index 00000000..04b24abc --- /dev/null +++ b/modules/gcp/databricks-workspace/tests/negative-serverless-destinations/main.tf @@ -0,0 +1,32 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { source = "databricks/databricks" } + google = { source = "hashicorp/google" } + } +} + +provider "google" { + project = "fixture-workspace" + region = "us-central1" +} + +provider "databricks" { + host = "https://accounts.gcp.databricks.com" + account_id = "00000000-0000-0000-0000-000000000000" +} + +# precondition fail: destinations without serverless_egress_mode=restricted +module "workspace" { + source = "../.." + + prefix = "fixture" + databricks_account_id = "00000000-0000-0000-0000-000000000000" + google_project = "fixture-workspace" + google_region = "us-central1" + + vpc_source = { spoke = "databricks_managed" } + + serverless_egress_mode = "full" + serverless_allowed_internet_destinations = ["example.com"] +} diff --git a/modules/gcp/databricks-workspace/tests/psc-isolated/main.tf b/modules/gcp/databricks-workspace/tests/psc-isolated/main.tf new file mode 100644 index 00000000..f1ad7e73 --- /dev/null +++ b/modules/gcp/databricks-workspace/tests/psc-isolated/main.tf @@ -0,0 +1,47 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { source = "databricks/databricks" } + google = { source = "hashicorp/google" } + } +} + +provider "google" { + project = "fixture-workspace" + region = "us-central1" +} + +provider "databricks" { + host = "https://accounts.gcp.databricks.com" + account_id = "00000000-0000-0000-0000-000000000000" +} + +module "workspace" { + source = "../.." + + prefix = "fixture" + databricks_account_id = "00000000-0000-0000-0000-000000000000" + google_project = "fixture-workspace" + google_region = "us-central1" + + vpc_source = { spoke = "create" } + spoke_vpc_cidr = "10.0.0.0/16" + subnet_cidr = "10.0.0.0/22" + + private_link_frontend = true + private_link_backend = true + private_access_only = true + restricted_egress = true + + spoke_vpc_google_project = "fixture-spoke" + hub_vpc_google_project = "fixture-hub" + is_spoke_vpc_shared = true + hub_vpc_cidr = "10.1.0.0/24" + psc_subnet_cidr = "10.0.255.0/28" + + serverless_egress_mode = "restricted" + serverless_allowed_internet_destinations = ["pypi.org"] + + cmek_managed_services_key_id = "projects/fixture-workspace/locations/us-central1/keyRings/fixture-kr/cryptoKeys/fixture-ms-key" + cmek_storage_key_id = "projects/fixture-workspace/locations/us-central1/keyRings/fixture-kr/cryptoKeys/fixture-storage-key" +} diff --git a/modules/gcp/databricks-workspace/tests/psc-no-peering/main.tf b/modules/gcp/databricks-workspace/tests/psc-no-peering/main.tf new file mode 100644 index 00000000..96edc277 --- /dev/null +++ b/modules/gcp/databricks-workspace/tests/psc-no-peering/main.tf @@ -0,0 +1,48 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { source = "databricks/databricks" } + google = { source = "hashicorp/google" } + } +} + +provider "google" { + project = "fixture-workspace" + region = "us-central1" +} + +provider "databricks" { + host = "https://accounts.gcp.databricks.com" + account_id = "00000000-0000-0000-0000-000000000000" +} + +module "workspace" { + source = "../.." + + prefix = "fixture" + databricks_account_id = "00000000-0000-0000-0000-000000000000" + google_project = "fixture-workspace" + google_region = "us-central1" + + vpc_source = { spoke = "create" } + spoke_vpc_cidr = "10.0.0.0/16" + subnet_cidr = "10.0.0.0/22" + + private_link_frontend = true + private_link_backend = true + private_access_only = true + restricted_egress = true + enable_hub_spoke_peering = false + + spoke_vpc_google_project = "fixture-spoke" + hub_vpc_google_project = "fixture-hub" + is_spoke_vpc_shared = true + hub_vpc_cidr = "10.1.0.0/24" + psc_subnet_cidr = "10.0.255.0/28" + + serverless_egress_mode = "restricted" + serverless_allowed_internet_destinations = ["pypi.org"] + + cmek_managed_services_key_id = "projects/fixture-workspace/locations/us-central1/keyRings/fixture-kr/cryptoKeys/fixture-ms-key" + cmek_storage_key_id = "projects/fixture-workspace/locations/us-central1/keyRings/fixture-kr/cryptoKeys/fixture-storage-key" +} diff --git a/modules/gcp/databricks-workspace/variables.tf b/modules/gcp/databricks-workspace/variables.tf new file mode 100644 index 00000000..cbcbc296 --- /dev/null +++ b/modules/gcp/databricks-workspace/variables.tf @@ -0,0 +1,210 @@ +# === Identity ============================================================ +variable "prefix" { + type = string + description = "Prefix used to name generated resources (e.g. \"acme\" produces \"acme-spoke-vpc-\")" +} + +variable "databricks_account_id" { + type = string + description = "Databricks account ID (GUID) where this workspace will be registered" +} + +variable "google_project" { + type = string + description = "GCP project ID hosting the workspace data plane" +} + +variable "google_region" { + type = string + description = "GCP region where the workspace will be deployed. When any private_link_* flag or restricted_egress is true, the region must be supported by Databricks PSC (see preconditions.tf)" +} + +variable "workspace_name" { + type = string + default = null + description = "Optional workspace name override. Defaults to \"prefix-ws-suffix\" when null" +} + +variable "tags" { + type = map(string) + default = {} + description = "Map of tags. Currently not propagated to child resources; reserved for future use" +} + +# === VPC source ========================================================== +variable "vpc_source" { + type = object({ + spoke = optional(string, "databricks_managed") + hub = optional(string) + }) + default = {} + description = "Where the workspace networks come from. spoke: databricks_managed (no networking module called), create (Terraform creates VPC + subnet + NAT), existing (data-source lookup of existing_vpc_name/existing_subnet_name). hub — only relevant when spoke is create or existing, and only consumed when restricted_egress=true: create (hub VPC + subnet created; hub_vpc_cidr required; the default when unset) or existing (lookup of existing_hub_vpc_name/existing_hub_subnet_name)" + validation { + condition = contains(["databricks_managed", "create", "existing"], var.vpc_source.spoke) + error_message = "vpc_source.spoke must be one of: databricks_managed, create, existing." + } + validation { + condition = var.vpc_source.hub == null || contains(["create", "existing"], var.vpc_source.hub) + error_message = "vpc_source.hub must be 'create' or 'existing' (or unset)." + } + validation { + condition = var.vpc_source.hub == null || contains(["create", "existing"], var.vpc_source.spoke) + error_message = "vpc_source.hub is only relevant when vpc_source.spoke is create or existing." + } +} + +# When vpc_source.spoke = "create" +variable "spoke_vpc_cidr" { + type = string + default = null + description = "CIDR of the spoke VPC address space (e.g. 10.0.0.0/16). Required when vpc_source.spoke=create; ignored otherwise" +} + +variable "subnet_cidr" { + type = string + default = null + description = "CIDR of the spoke subnet primary range (e.g. 10.0.0.0/22). Required when vpc_source.spoke=create" +} + +# When vpc_source.spoke = "existing" +variable "existing_vpc_name" { + type = string + default = null + description = "Name of the pre-existing VPC to use. Required when vpc_source.spoke=existing" +} + +variable "existing_subnet_name" { + type = string + default = null + description = "Name of the pre-existing subnet to use (must be in google_region). Required when vpc_source.spoke=existing" +} + +# === Connectivity feature flags ========================================== +variable "private_link_frontend" { + type = bool + default = false + description = "Create the frontend (workspace UI/API) PSC endpoint and a frontend databricks_mws_vpc_endpoint. On GCP both flags must be enabled together (see preconditions.tf)" +} + +variable "private_link_backend" { + type = bool + default = false + description = "Create the backend (SCC, data plane) PSC endpoint and a backend databricks_mws_vpc_endpoint. On GCP both flags must be enabled together (see preconditions.tf)" +} + +variable "private_access_only" { + type = bool + default = false + description = "Create databricks_mws_private_access_settings with public_access_enabled=false. Workspace becomes reachable only through PSC endpoints" +} + +variable "restricted_egress" { + type = bool + default = false + description = "Create hub VPC + bidirectional peering + deny-egress firewall + private DNS zones. Requires vpc_source.spoke=create and at least one private_link_* flag" +} + +# === Required when restricted_egress = true ============================== +variable "hub_vpc_google_project" { + type = string + default = null + description = "GCP project hosting the hub VPC. Required when restricted_egress=true" +} + +variable "spoke_vpc_google_project" { + type = string + default = null + description = "GCP project hosting the spoke VPC. Defaults to google_project when null" +} + +variable "is_spoke_vpc_shared" { + type = bool + default = false + description = "If true and the spoke VPC project differs from the workspace project, bind the spoke project as a Shared-VPC host and the workspace project as a service project. Works with or without restricted_egress" +} + +variable "hub_vpc_cidr" { + type = string + default = null + description = "CIDR of the hub subnet (e.g. 10.1.0.0/24). Required when restricted_egress=true and vpc_source.hub=create" +} + +variable "enable_hub_spoke_peering" { + type = bool + default = true + description = "Create the bidirectional VPC peering between hub and spoke. Disable when hub-spoke connectivity is provided by other means (e.g. Shared VPC or an existing transit). Cloud DNS peering zones do not depend on it. Only takes effect when the hub is enabled" +} + +variable "existing_hub_vpc_name" { + type = string + default = null + description = "Name of the pre-existing hub VPC. Required when vpc_source.hub=existing" +} + +variable "existing_hub_subnet_name" { + type = string + default = null + description = "Name of the pre-existing hub subnet (must be in google_region). Required when vpc_source.hub=existing" +} + +variable "psc_subnet_cidr" { + type = string + default = null + description = "CIDR of the dedicated PSC subnet in the spoke VPC (e.g. 10.0.255.0/28). Required when restricted_egress=true or any private_link_* flag is true" +} + +variable "hive_metastore_ip" { + type = string + default = null + description = "Regional legacy Hive metastore IP. When set, an egress allow rule (tcp/3306) is created under restricted egress; when null, no rule is created. Workspaces using Unity Catalog (the default) do not need this. Regional IPs: https://docs.databricks.com/gcp/en/resources/ip-domain-region" + validation { + condition = var.hive_metastore_ip == null || can(cidrnetmask("${var.hive_metastore_ip}/32")) + error_message = "hive_metastore_ip must be a valid IPv4 address, or null to skip the rule." + } +} + +# === Serverless egress control =========================================== +variable "serverless_egress_mode" { + type = string + default = "unmanaged" + description = "Serverless egress control. unmanaged: no network policy resources; full: policy with FULL_ACCESS; restricted: deny-by-default policy allowing only the listed destinations. Requires the workspace to be on the Enterprise tier" + validation { + condition = contains(["unmanaged", "full", "restricted"], var.serverless_egress_mode) + error_message = "serverless_egress_mode must be one of: unmanaged, full, restricted." + } +} + +variable "serverless_allowed_internet_destinations" { + type = list(string) + default = [] + description = "FQDNs serverless workloads may reach when serverless_egress_mode=restricted (max 100)" +} + +variable "serverless_allowed_storage_destinations" { + type = list(string) + default = [] + description = "GCS bucket names serverless workloads may reach when serverless_egress_mode=restricted (max 100); region is taken from google_region" +} + +variable "serverless_egress_enforcement" { + type = string + default = "enforced" + description = "enforced: violations are blocked; dry_run: violations are only logged (use to evaluate a policy before enforcing)" + validation { + condition = contains(["enforced", "dry_run"], var.serverless_egress_enforcement) + error_message = "serverless_egress_enforcement must be one of: enforced, dry_run." + } +} + +# === Customer-managed keys (CMEK) ======================================= +variable "cmek_managed_services_key_id" { + type = string + default = null + description = "Cloud KMS key resource ID for managed-services CMEK (control-plane data: notebooks, secrets, queries). Null disables. The principal running Terraform needs cloudkms.cryptoKeys.getIamPolicy and setIamPolicy on the key - Databricks sets the key's IAM policy at workspace creation. Enterprise tier; set at creation only. The key must exist before plan (a key created in the same configuration makes the count unknown and fails plan)" +} + +variable "cmek_storage_key_id" { + type = string + default = null + description = "Cloud KMS key resource ID for workspace-storage CMEK (GCS buckets and GCE persistent disks). Null disables. Same permission and tier requirements as cmek_managed_services_key_id; set at creation only. The key must exist before plan (a key created in the same configuration makes the count unknown and fails plan)" +} diff --git a/modules/gcp/databricks-workspace/versions.tf b/modules/gcp/databricks-workspace/versions.tf new file mode 100644 index 00000000..d5367540 --- /dev/null +++ b/modules/gcp/databricks-workspace/versions.tf @@ -0,0 +1,17 @@ +terraform { + required_version = ">= 1.5" + required_providers { + google = { + source = "hashicorp/google" + version = ">= 6.0" + } + databricks = { + source = "databricks/databricks" + version = ">= 1.81.1" + } + random = { + source = "hashicorp/random" + version = ">= 3.0" + } + } +} diff --git a/modules/gcp/dns/Makefile b/modules/gcp/dns/Makefile new file mode 100644 index 00000000..17b32ec8 --- /dev/null +++ b/modules/gcp/dns/Makefile @@ -0,0 +1,7 @@ +.PHONY: docs test_docs + +docs: + terraform-docs -c ../../../.terraform-docs.yml . + +test_docs: + terraform-docs -c ../../../.terraform-docs.yml --output-check . diff --git a/modules/gcp/dns/README.md b/modules/gcp/dns/README.md new file mode 100644 index 00000000..466c6e1f --- /dev/null +++ b/modules/gcp/dns/README.md @@ -0,0 +1,93 @@ +# modules/gcp/dns + +Private DNS zones (hub + spoke) used with restricted-egress workspaces. + +## Usage + +Typically called by `modules/gcp/databricks-workspace` (the composer) when `restricted_egress=true`. Direct consumption is unusual; this module is terminal (no outputs). + +```hcl +module "dns" { + source = "github.com/databricks/terraform-databricks-examples//modules/gcp/dns" + + prefix = "acme" + google_region = "us-central1" + + hub_vpc_id = module.network.hub_vpc_id + hub_vpc_self_link = module.network.hub_vpc_self_link + hub_vpc_google_project = "my-hub-project" + + spoke_vpc_id = module.network.spoke_vpc_id + spoke_vpc_self_link = module.network.spoke_vpc_self_link + spoke_vpc_google_project = "my-spoke-project" + + workspace_url = module.workspace.workspace_url + + frontend_psc_ip_spoke = module.private_connectivity.frontend_psc_ip_spoke + frontend_psc_ip_hub = module.private_connectivity.frontend_psc_ip_hub + backend_psc_ip_spoke = module.private_connectivity.backend_psc_ip_spoke +} +``` + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.5 | +| [google](#requirement\_google) | >= 6.0 | + +## Providers + +| Name | Version | +|------|---------| +| [google](#provider\_google) | 7.39.0 | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [google_dns_managed_zone.gcr](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_managed_zone) | resource | +| [google_dns_managed_zone.google_apis](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_managed_zone) | resource | +| [google_dns_managed_zone.hub_databricks](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_managed_zone) | resource | +| [google_dns_managed_zone.pkg_dev](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_managed_zone) | resource | +| [google_dns_managed_zone.spoke_databricks](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_managed_zone) | resource | +| [google_dns_managed_zone.spoke_peering_gcr](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_managed_zone) | resource | +| [google_dns_managed_zone.spoke_peering_google_apis](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_managed_zone) | resource | +| [google_dns_managed_zone.spoke_peering_pkg_dev](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_managed_zone) | resource | +| [google_dns_record_set.gcr_a](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | +| [google_dns_record_set.gcr_cname](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | +| [google_dns_record_set.google_apis_a](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | +| [google_dns_record_set.google_apis_cname](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | +| [google_dns_record_set.hub_dp](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | +| [google_dns_record_set.hub_psc_auth](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | +| [google_dns_record_set.hub_workspace_url](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | +| [google_dns_record_set.pkg_dev_a](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | +| [google_dns_record_set.pkg_dev_cname](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | +| [google_dns_record_set.spoke_dp](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | +| [google_dns_record_set.spoke_tunnel](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | +| [google_dns_record_set.spoke_workspace_url](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/dns_record_set) | resource | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [backend\_psc\_ip\_spoke](#input\_backend\_psc\_ip\_spoke) | Spoke-side backend (SCC) PSC endpoint IP (used in the spoke tunnel..gcp.databricks.com A record) | `string` | n/a | yes | +| [frontend\_psc\_ip\_spoke](#input\_frontend\_psc\_ip\_spoke) | Spoke-side frontend PSC endpoint IP (used in the spoke gcp.databricks.com A records) | `string` | n/a | yes | +| [google\_region](#input\_google\_region) | GCP region (used in the spoke tunnel DNS record name) | `string` | n/a | yes | +| [hub\_vpc\_google\_project](#input\_hub\_vpc\_google\_project) | GCP project hosting the hub VPC (used for the hub DNS zones) | `string` | n/a | yes | +| [hub\_vpc\_id](#input\_hub\_vpc\_id) | ID of the hub VPC (DNS zones with this VPC's visibility) | `string` | n/a | yes | +| [prefix](#input\_prefix) | Prefix used to name generated DNS managed zones | `string` | n/a | yes | +| [spoke\_vpc\_google\_project](#input\_spoke\_vpc\_google\_project) | GCP project hosting the spoke VPC (used for the spoke DNS zone) | `string` | n/a | yes | +| [spoke\_vpc\_id](#input\_spoke\_vpc\_id) | ID of the spoke VPC (DNS zone with this VPC's visibility) | `string` | n/a | yes | +| [workspace\_url](#input\_workspace\_url) | Workspace URL from databricks\_mws\_workspaces; used to extract the workspace DNS ID via regex | `string` | n/a | yes | +| [frontend\_psc\_ip\_hub](#input\_frontend\_psc\_ip\_hub) | Hub-side frontend PSC endpoint IP (used in the hub gcp.databricks.com A records) | `string` | `null` | no | + +## Outputs + +No outputs. + diff --git a/modules/gcp/dns/hub.tf b/modules/gcp/dns/hub.tf new file mode 100644 index 00000000..e3103e89 --- /dev/null +++ b/modules/gcp/dns/hub.tf @@ -0,0 +1,140 @@ +# === gcp.databricks.com (hub) ============================================ +resource "google_dns_managed_zone" "hub_databricks" { + name = "${var.prefix}-hub-gcp-databricks-com" + project = var.hub_vpc_google_project + dns_name = "gcp.databricks.com." + description = "Private DNS zone for Databricks PSC management" + visibility = "private" + + private_visibility_config { + networks { + network_url = var.hub_vpc_id + } + } +} + +resource "google_dns_record_set" "hub_workspace_url" { + name = "${local.workspace_dns_id}.${google_dns_managed_zone.hub_databricks.dns_name}" + project = var.hub_vpc_google_project + managed_zone = google_dns_managed_zone.hub_databricks.name + type = "A" + ttl = 300 + rrdatas = [var.frontend_psc_ip_hub] +} + +resource "google_dns_record_set" "hub_psc_auth" { + name = "${var.google_region}.psc-auth.${google_dns_managed_zone.hub_databricks.dns_name}" + project = var.hub_vpc_google_project + managed_zone = google_dns_managed_zone.hub_databricks.name + type = "A" + ttl = 300 + rrdatas = [var.frontend_psc_ip_hub] +} + +resource "google_dns_record_set" "hub_dp" { + name = "dp-${local.workspace_dns_id}.${google_dns_managed_zone.hub_databricks.dns_name}" + project = var.hub_vpc_google_project + managed_zone = google_dns_managed_zone.hub_databricks.name + type = "A" + ttl = 300 + rrdatas = [var.frontend_psc_ip_hub] +} + +# === gcr.io ============================================================== +resource "google_dns_managed_zone" "gcr" { + name = "${var.prefix}-gcr-io" + project = var.hub_vpc_google_project + dns_name = "gcr.io." + description = "Private DNS zone for GCR private resolution" + visibility = "private" + + private_visibility_config { + networks { + network_url = var.hub_vpc_id + } + } +} + +resource "google_dns_record_set" "gcr_cname" { + name = "*.${google_dns_managed_zone.gcr.dns_name}" + project = var.hub_vpc_google_project + managed_zone = google_dns_managed_zone.gcr.name + type = "CNAME" + ttl = 300 + rrdatas = ["gcr.io."] +} + +resource "google_dns_record_set" "gcr_a" { + name = google_dns_managed_zone.gcr.dns_name + project = var.hub_vpc_google_project + managed_zone = google_dns_managed_zone.gcr.name + type = "A" + ttl = 300 + rrdatas = ["199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"] +} + +# === googleapis.com ====================================================== +resource "google_dns_managed_zone" "google_apis" { + name = "${var.prefix}-google-apis" + project = var.hub_vpc_google_project + dns_name = "googleapis.com." + description = "Private DNS zone for Google APIs resolution" + visibility = "private" + + private_visibility_config { + networks { + network_url = var.hub_vpc_id + } + } +} + +resource "google_dns_record_set" "google_apis_cname" { + name = "*.${google_dns_managed_zone.google_apis.dns_name}" + project = var.hub_vpc_google_project + managed_zone = google_dns_managed_zone.google_apis.name + type = "CNAME" + ttl = 300 + rrdatas = ["restricted.googleapis.com."] +} + +resource "google_dns_record_set" "google_apis_a" { + name = "restricted.${google_dns_managed_zone.google_apis.dns_name}" + project = var.hub_vpc_google_project + managed_zone = google_dns_managed_zone.google_apis.name + type = "A" + ttl = 300 + rrdatas = ["199.36.153.4", "199.36.153.5", "199.36.153.6", "199.36.153.7"] +} + +# === pkg.dev ============================================================= +resource "google_dns_managed_zone" "pkg_dev" { + name = "${var.prefix}-pkg-dev" + project = var.hub_vpc_google_project + dns_name = "pkg.dev." + description = "Private DNS zone for Go Packages resolution" + visibility = "private" + + private_visibility_config { + networks { + network_url = var.hub_vpc_id + } + } +} + +resource "google_dns_record_set" "pkg_dev_cname" { + name = "*.${google_dns_managed_zone.pkg_dev.dns_name}" + project = var.hub_vpc_google_project + managed_zone = google_dns_managed_zone.pkg_dev.name + type = "CNAME" + ttl = 300 + rrdatas = ["pkg.dev."] +} + +resource "google_dns_record_set" "pkg_dev_a" { + name = google_dns_managed_zone.pkg_dev.dns_name + project = var.hub_vpc_google_project + managed_zone = google_dns_managed_zone.pkg_dev.name + type = "A" + ttl = 300 + rrdatas = ["199.36.153.8", "199.36.153.9", "199.36.153.10", "199.36.153.11"] +} diff --git a/modules/gcp/dns/locals.tf b/modules/gcp/dns/locals.tf new file mode 100644 index 00000000..9e2c4fd2 --- /dev/null +++ b/modules/gcp/dns/locals.tf @@ -0,0 +1,4 @@ +locals { + # Regex extracts the workspace DNS id (numeric.numeric) from the URL. + workspace_dns_id = regex("[0-9]+\\.[0-9]+", var.workspace_url) +} diff --git a/modules/gcp/dns/outputs.tf b/modules/gcp/dns/outputs.tf new file mode 100644 index 00000000..19cbc3d5 --- /dev/null +++ b/modules/gcp/dns/outputs.tf @@ -0,0 +1 @@ +# This module has no outputs; DNS records are terminal. diff --git a/modules/gcp/dns/spoke.tf b/modules/gcp/dns/spoke.tf new file mode 100644 index 00000000..6dbf8b6d --- /dev/null +++ b/modules/gcp/dns/spoke.tf @@ -0,0 +1,105 @@ +# === gcp.databricks.com (spoke) ========================================== +resource "google_dns_managed_zone" "spoke_databricks" { + name = "${var.prefix}-spoke-gcp-databricks-com" + project = var.spoke_vpc_google_project + dns_name = "gcp.databricks.com." + description = "Private DNS zone for Databricks PSC management" + visibility = "private" + + private_visibility_config { + networks { + network_url = var.spoke_vpc_id + } + } +} + +resource "google_dns_record_set" "spoke_workspace_url" { + name = "${local.workspace_dns_id}.${google_dns_managed_zone.spoke_databricks.dns_name}" + project = var.spoke_vpc_google_project + managed_zone = google_dns_managed_zone.spoke_databricks.name + type = "A" + ttl = 300 + rrdatas = [var.frontend_psc_ip_spoke] +} + +resource "google_dns_record_set" "spoke_dp" { + name = "dp-${local.workspace_dns_id}.${google_dns_managed_zone.spoke_databricks.dns_name}" + project = var.spoke_vpc_google_project + managed_zone = google_dns_managed_zone.spoke_databricks.name + type = "A" + ttl = 300 + rrdatas = [var.frontend_psc_ip_spoke] +} + +resource "google_dns_record_set" "spoke_tunnel" { + name = "tunnel.${var.google_region}.${google_dns_managed_zone.spoke_databricks.dns_name}" + project = var.spoke_vpc_google_project + managed_zone = google_dns_managed_zone.spoke_databricks.name + type = "A" + ttl = 300 + rrdatas = [var.backend_psc_ip_spoke] +} + +# === Peering zones (spoke → hub) ========================================= +# Private zones do not propagate over VPC peering. The hub hosts the +# record-bearing zones for googleapis.com / gcr.io / pkg.dev; these peering +# zones make them resolvable from the spoke. +resource "google_dns_managed_zone" "spoke_peering_google_apis" { + name = "${var.prefix}-peering-google-apis" + project = var.spoke_vpc_google_project + dns_name = "googleapis.com." + description = "Peering DNS zone delegating googleapis.com resolution to the hub VPC" + visibility = "private" + + private_visibility_config { + networks { + network_url = var.spoke_vpc_id + } + } + + peering_config { + target_network { + network_url = var.hub_vpc_id + } + } +} + +resource "google_dns_managed_zone" "spoke_peering_gcr" { + name = "${var.prefix}-peering-gcr" + project = var.spoke_vpc_google_project + dns_name = "gcr.io." + description = "Peering DNS zone delegating gcr.io resolution to the hub VPC" + visibility = "private" + + private_visibility_config { + networks { + network_url = var.spoke_vpc_id + } + } + + peering_config { + target_network { + network_url = var.hub_vpc_id + } + } +} + +resource "google_dns_managed_zone" "spoke_peering_pkg_dev" { + name = "${var.prefix}-peering-pkg-dev" + project = var.spoke_vpc_google_project + dns_name = "pkg.dev." + description = "Peering DNS zone delegating pkg.dev resolution to the hub VPC" + visibility = "private" + + private_visibility_config { + networks { + network_url = var.spoke_vpc_id + } + } + + peering_config { + target_network { + network_url = var.hub_vpc_id + } + } +} diff --git a/modules/gcp/dns/tests/hub-and-spoke/main.tf b/modules/gcp/dns/tests/hub-and-spoke/main.tf new file mode 100644 index 00000000..bbaee42a --- /dev/null +++ b/modules/gcp/dns/tests/hub-and-spoke/main.tf @@ -0,0 +1,27 @@ +terraform { + required_version = ">= 1.5" +} + +provider "google" { + project = "fixture-spoke" + region = "us-central1" +} + +module "dns" { + source = "../.." + + prefix = "fixture" + google_region = "us-central1" + + hub_vpc_id = "projects/fixture-hub/global/networks/hub-vpc" + hub_vpc_google_project = "fixture-hub" + + spoke_vpc_id = "projects/fixture-spoke/global/networks/spoke-vpc" + spoke_vpc_google_project = "fixture-spoke" + + workspace_url = "https://1234567890123456.7.gcp.databricks.com" + + frontend_psc_ip_spoke = "10.0.255.4" + frontend_psc_ip_hub = "10.1.0.10" + backend_psc_ip_spoke = "10.0.255.5" +} diff --git a/modules/gcp/dns/variables.tf b/modules/gcp/dns/variables.tf new file mode 100644 index 00000000..e949ff14 --- /dev/null +++ b/modules/gcp/dns/variables.tf @@ -0,0 +1,54 @@ +variable "prefix" { + type = string + description = "Prefix used to name generated DNS managed zones" +} + +variable "google_region" { + type = string + description = "GCP region (used in the spoke tunnel DNS record name)" +} + +# Hub +variable "hub_vpc_id" { + type = string + description = "ID of the hub VPC (DNS zones with this VPC's visibility)" +} + +variable "hub_vpc_google_project" { + type = string + description = "GCP project hosting the hub VPC (used for the hub DNS zones)" +} + +# Spoke +variable "spoke_vpc_id" { + type = string + description = "ID of the spoke VPC (DNS zone with this VPC's visibility)" +} + +variable "spoke_vpc_google_project" { + type = string + description = "GCP project hosting the spoke VPC (used for the spoke DNS zone)" +} + +# Workspace +variable "workspace_url" { + type = string + description = "Workspace URL from databricks_mws_workspaces; used to extract the workspace DNS ID via regex" +} + +# PSC IPs +variable "frontend_psc_ip_spoke" { + type = string + description = "Spoke-side frontend PSC endpoint IP (used in the spoke gcp.databricks.com A records)" +} + +variable "frontend_psc_ip_hub" { + type = string + default = null + description = "Hub-side frontend PSC endpoint IP (used in the hub gcp.databricks.com A records)" +} + +variable "backend_psc_ip_spoke" { + type = string + description = "Spoke-side backend (SCC) PSC endpoint IP (used in the spoke tunnel..gcp.databricks.com A record)" +} diff --git a/modules/gcp/dns/versions.tf b/modules/gcp/dns/versions.tf new file mode 100644 index 00000000..8b5ec227 --- /dev/null +++ b/modules/gcp/dns/versions.tf @@ -0,0 +1,9 @@ +terraform { + required_version = ">= 1.5" + required_providers { + google = { + source = "hashicorp/google" + version = ">= 6.0" + } + } +} diff --git a/modules/gcp/network/Makefile b/modules/gcp/network/Makefile new file mode 100644 index 00000000..17b32ec8 --- /dev/null +++ b/modules/gcp/network/Makefile @@ -0,0 +1,7 @@ +.PHONY: docs test_docs + +docs: + terraform-docs -c ../../../.terraform-docs.yml . + +test_docs: + terraform-docs -c ../../../.terraform-docs.yml --output-check . diff --git a/modules/gcp/network/README.md b/modules/gcp/network/README.md new file mode 100644 index 00000000..60ee2139 --- /dev/null +++ b/modules/gcp/network/README.md @@ -0,0 +1,99 @@ +# modules/gcp/network + +VPC, subnet, router, NAT, peering, and Shared-VPC binding for the Databricks GCP composer. + +## Usage + +Typically called by `modules/gcp/databricks-workspace` (the composer). Direct consumption is supported but unusual; you'll need to wire the outputs yourself. + +```hcl +module "network" { + source = "github.com/databricks/terraform-databricks-examples//modules/gcp/network" + + prefix = "acme" + suffix = "abc123" + google_region = "us-central1" + vpc_source = "create" + spoke_vpc_google_project = "my-project" + spoke_vpc_cidr = "10.0.0.0/16" + subnet_cidr = "10.0.0.0/22" +} +``` + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.5 | +| [google](#requirement\_google) | >= 6.0 | + +## Providers + +| Name | Version | +|------|---------| +| [google](#provider\_google) | 7.39.0 | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [google_compute_network.hub_vpc](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_network) | resource | +| [google_compute_network.spoke_vpc](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_network) | resource | +| [google_compute_network_peering.hub_to_spoke](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_network_peering) | resource | +| [google_compute_network_peering.spoke_to_hub](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_network_peering) | resource | +| [google_compute_router.router](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_router) | resource | +| [google_compute_router_nat.nat](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_router_nat) | resource | +| [google_compute_shared_vpc_host_project.host](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_shared_vpc_host_project) | resource | +| [google_compute_shared_vpc_service_project.service](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_shared_vpc_service_project) | resource | +| [google_compute_subnetwork.hub_subnet](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_subnetwork) | resource | +| [google_compute_subnetwork.spoke_subnet](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_subnetwork) | resource | +| [google_compute_network.existing_hub](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/compute_network) | data source | +| [google_compute_network.existing_spoke](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/compute_network) | data source | +| [google_compute_subnetwork.existing_hub_subnet](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/compute_subnetwork) | data source | +| [google_compute_subnetwork.existing_spoke_subnet](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/compute_subnetwork) | data source | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [google\_region](#input\_google\_region) | GCP region for all network resources | `string` | n/a | yes | +| [prefix](#input\_prefix) | Prefix for generated resource names | `string` | n/a | yes | +| [spoke\_vpc\_google\_project](#input\_spoke\_vpc\_google\_project) | GCP project hosting the spoke VPC | `string` | n/a | yes | +| [suffix](#input\_suffix) | Random suffix passed by the composer for uniqueness | `string` | n/a | yes | +| [vpc\_source](#input\_vpc\_source) | Either 'create' (Terraform creates a VPC) or 'existing' (data-source lookup) | `string` | n/a | yes | +| [enable\_hub](#input\_enable\_hub) | Enable the hub half of the topology: hub VPC/subnet (created or looked up per hub\_vpc\_source) and hub-spoke peering. Composer passes restricted\_egress | `bool` | `false` | no | +| [enable\_hub\_spoke\_peering](#input\_enable\_hub\_spoke\_peering) | Create the bidirectional VPC peering between hub and spoke. Disable when hub-spoke connectivity is provided by other means (e.g. Shared VPC or an existing transit). Cloud DNS peering zones do not depend on it. Only takes effect when the hub is enabled | `bool` | `true` | no | +| [enable\_nat](#input\_enable\_nat) | Create Cloud Router + NAT for internet egress. The composer disables this under restricted\_egress, where no internet egress path may exist | `bool` | `true` | no | +| [existing\_hub\_subnet\_name](#input\_existing\_hub\_subnet\_name) | Name of the pre-existing hub subnet (must be in google\_region). Required when hub\_vpc\_source=existing | `string` | `null` | no | +| [existing\_hub\_vpc\_name](#input\_existing\_hub\_vpc\_name) | Name of the pre-existing hub VPC. Required when hub\_vpc\_source=existing | `string` | `null` | no | +| [existing\_subnet\_name](#input\_existing\_subnet\_name) | Name of pre-existing subnet (required when vpc\_source=existing) | `string` | `null` | no | +| [existing\_vpc\_name](#input\_existing\_vpc\_name) | Name of pre-existing VPC (required when vpc\_source=existing) | `string` | `null` | no | +| [hub\_vpc\_cidr](#input\_hub\_vpc\_cidr) | CIDR for the hub subnet (required when enable\_hub=true) | `string` | `null` | no | +| [hub\_vpc\_google\_project](#input\_hub\_vpc\_google\_project) | GCP project hosting the hub VPC (required when enable\_hub=true) | `string` | `null` | no | +| [hub\_vpc\_source](#input\_hub\_vpc\_source) | Where the hub VPC comes from when the hub is enabled. create: Terraform creates the hub VPC and subnet (hub\_vpc\_cidr required); existing: data-source lookup of existing\_hub\_vpc\_name/existing\_hub\_subnet\_name in hub\_vpc\_google\_project | `string` | `"create"` | no | +| [is\_spoke\_vpc\_shared](#input\_is\_spoke\_vpc\_shared) | If true, bind the spoke VPC's project as a Shared-VPC host and the workspace project as a service project | `bool` | `false` | no | +| [subnet\_cidr](#input\_subnet\_cidr) | CIDR for the spoke subnet (required when vpc\_source=create) | `string` | `null` | no | +| [subnet\_name](#input\_subnet\_name) | Override for spoke subnet name (default: "{prefix}-subnet-{suffix}") | `string` | `null` | no | +| [workspace\_google\_project](#input\_workspace\_google\_project) | Workspace project (used for Shared-VPC service binding) | `string` | `null` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| [hub\_subnet\_name](#output\_hub\_subnet\_name) | Name of the hub subnet (null when the hub is disabled) | +| [hub\_vpc\_id](#output\_hub\_vpc\_id) | ID of the hub VPC (null when the hub is disabled) | +| [hub\_vpc\_name](#output\_hub\_vpc\_name) | Name of the hub VPC (null when the hub is disabled) | +| [hub\_vpc\_self\_link](#output\_hub\_vpc\_self\_link) | Self-link of the hub VPC (null when the hub is disabled) | +| [nat\_id](#output\_nat\_id) | ID of the Cloud NAT (null when vpc\_source=existing or enable\_nat=false) | +| [spoke\_subnet\_id](#output\_spoke\_subnet\_id) | ID of the spoke subnet | +| [spoke\_subnet\_name](#output\_spoke\_subnet\_name) | Name of the spoke subnet | +| [spoke\_subnet\_self\_link](#output\_spoke\_subnet\_self\_link) | Self-link of the spoke subnet | +| [spoke\_vpc\_id](#output\_spoke\_vpc\_id) | ID of the spoke VPC | +| [spoke\_vpc\_name](#output\_spoke\_vpc\_name) | Name of the spoke VPC | +| [spoke\_vpc\_self\_link](#output\_spoke\_vpc\_self\_link) | Self-link of the spoke VPC | + diff --git a/modules/gcp/network/data.tf b/modules/gcp/network/data.tf new file mode 100644 index 00000000..98ddff48 --- /dev/null +++ b/modules/gcp/network/data.tf @@ -0,0 +1,29 @@ +data "google_compute_network" "existing_spoke" { + count = local.use_existing_spoke ? 1 : 0 + + name = var.existing_vpc_name + project = var.spoke_vpc_google_project +} + +data "google_compute_subnetwork" "existing_spoke_subnet" { + count = local.use_existing_spoke ? 1 : 0 + + name = var.existing_subnet_name + project = var.spoke_vpc_google_project + region = var.google_region +} + +data "google_compute_network" "existing_hub" { + count = local.use_existing_hub ? 1 : 0 + + name = var.existing_hub_vpc_name + project = var.hub_vpc_google_project +} + +data "google_compute_subnetwork" "existing_hub_subnet" { + count = local.use_existing_hub ? 1 : 0 + + name = var.existing_hub_subnet_name + project = var.hub_vpc_google_project + region = var.google_region +} diff --git a/modules/gcp/network/locals.tf b/modules/gcp/network/locals.tf new file mode 100644 index 00000000..9a0c48ed --- /dev/null +++ b/modules/gcp/network/locals.tf @@ -0,0 +1,9 @@ +locals { + create_spoke = var.vpc_source == "create" + use_existing_spoke = var.vpc_source == "existing" + + create_hub_vpc = var.enable_hub && var.hub_vpc_source == "create" + use_existing_hub = var.enable_hub && var.hub_vpc_source == "existing" + + subnet_name = coalesce(var.subnet_name, "${var.prefix}-subnet-${var.suffix}") +} diff --git a/modules/gcp/network/nat.tf b/modules/gcp/network/nat.tf new file mode 100644 index 00000000..9457aa14 --- /dev/null +++ b/modules/gcp/network/nat.tf @@ -0,0 +1,19 @@ +resource "google_compute_router" "router" { + count = local.create_spoke && var.enable_nat ? 1 : 0 + + name = "${var.prefix}-router-${var.suffix}" + project = var.spoke_vpc_google_project + region = var.google_region + network = google_compute_network.spoke_vpc[0].id +} + +resource "google_compute_router_nat" "nat" { + count = local.create_spoke && var.enable_nat ? 1 : 0 + + name = "${var.prefix}-nat-${var.suffix}" + project = var.spoke_vpc_google_project + router = google_compute_router.router[0].name + region = var.google_region + nat_ip_allocate_option = "AUTO_ONLY" + source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES" +} diff --git a/modules/gcp/network/outputs.tf b/modules/gcp/network/outputs.tf new file mode 100644 index 00000000..1605c088 --- /dev/null +++ b/modules/gcp/network/outputs.tf @@ -0,0 +1,74 @@ +output "spoke_vpc_id" { + value = local.create_spoke ? google_compute_network.spoke_vpc[0].id : ( + local.use_existing_spoke ? data.google_compute_network.existing_spoke[0].id : null + ) + description = "ID of the spoke VPC" +} + +output "spoke_vpc_name" { + value = local.create_spoke ? google_compute_network.spoke_vpc[0].name : ( + local.use_existing_spoke ? data.google_compute_network.existing_spoke[0].name : null + ) + description = "Name of the spoke VPC" +} + +output "spoke_vpc_self_link" { + value = local.create_spoke ? google_compute_network.spoke_vpc[0].self_link : ( + local.use_existing_spoke ? data.google_compute_network.existing_spoke[0].self_link : null + ) + description = "Self-link of the spoke VPC" +} + +output "spoke_subnet_id" { + value = local.create_spoke ? google_compute_subnetwork.spoke_subnet[0].id : ( + local.use_existing_spoke ? data.google_compute_subnetwork.existing_spoke_subnet[0].id : null + ) + description = "ID of the spoke subnet" +} + +output "spoke_subnet_name" { + value = local.create_spoke ? google_compute_subnetwork.spoke_subnet[0].name : ( + local.use_existing_spoke ? data.google_compute_subnetwork.existing_spoke_subnet[0].name : null + ) + description = "Name of the spoke subnet" +} + +output "spoke_subnet_self_link" { + value = local.create_spoke ? google_compute_subnetwork.spoke_subnet[0].self_link : ( + local.use_existing_spoke ? data.google_compute_subnetwork.existing_spoke_subnet[0].self_link : null + ) + description = "Self-link of the spoke subnet" +} + +output "nat_id" { + value = local.create_spoke && var.enable_nat ? google_compute_router_nat.nat[0].id : null + description = "ID of the Cloud NAT (null when vpc_source=existing or enable_nat=false)" +} + +output "hub_vpc_id" { + value = local.create_hub_vpc ? google_compute_network.hub_vpc[0].id : ( + local.use_existing_hub ? data.google_compute_network.existing_hub[0].id : null + ) + description = "ID of the hub VPC (null when the hub is disabled)" +} + +output "hub_vpc_name" { + value = local.create_hub_vpc ? google_compute_network.hub_vpc[0].name : ( + local.use_existing_hub ? data.google_compute_network.existing_hub[0].name : null + ) + description = "Name of the hub VPC (null when the hub is disabled)" +} + +output "hub_vpc_self_link" { + value = local.create_hub_vpc ? google_compute_network.hub_vpc[0].self_link : ( + local.use_existing_hub ? data.google_compute_network.existing_hub[0].self_link : null + ) + description = "Self-link of the hub VPC (null when the hub is disabled)" +} + +output "hub_subnet_name" { + value = local.create_hub_vpc ? google_compute_subnetwork.hub_subnet[0].name : ( + local.use_existing_hub ? data.google_compute_subnetwork.existing_hub_subnet[0].name : null + ) + description = "Name of the hub subnet (null when the hub is disabled)" +} diff --git a/modules/gcp/network/peering.tf b/modules/gcp/network/peering.tf new file mode 100644 index 00000000..c8181a93 --- /dev/null +++ b/modules/gcp/network/peering.tf @@ -0,0 +1,15 @@ +resource "google_compute_network_peering" "hub_to_spoke" { + count = var.enable_hub && var.enable_hub_spoke_peering ? 1 : 0 + + name = "${var.prefix}-hub-spoke-${var.suffix}" + network = local.create_hub_vpc ? google_compute_network.hub_vpc[0].self_link : data.google_compute_network.existing_hub[0].self_link + peer_network = local.create_spoke ? google_compute_network.spoke_vpc[0].self_link : data.google_compute_network.existing_spoke[0].self_link +} + +resource "google_compute_network_peering" "spoke_to_hub" { + count = var.enable_hub && var.enable_hub_spoke_peering ? 1 : 0 + + name = "${var.prefix}-spoke-hub-${var.suffix}" + network = local.create_spoke ? google_compute_network.spoke_vpc[0].self_link : data.google_compute_network.existing_spoke[0].self_link + peer_network = local.create_hub_vpc ? google_compute_network.hub_vpc[0].self_link : data.google_compute_network.existing_hub[0].self_link +} diff --git a/modules/gcp/network/shared-vpc.tf b/modules/gcp/network/shared-vpc.tf new file mode 100644 index 00000000..337186c5 --- /dev/null +++ b/modules/gcp/network/shared-vpc.tf @@ -0,0 +1,12 @@ +resource "google_compute_shared_vpc_host_project" "host" { + count = var.is_spoke_vpc_shared && var.workspace_google_project != var.spoke_vpc_google_project ? 1 : 0 + + project = var.spoke_vpc_google_project +} + +resource "google_compute_shared_vpc_service_project" "service" { + count = var.is_spoke_vpc_shared && var.workspace_google_project != var.spoke_vpc_google_project ? 1 : 0 + + host_project = google_compute_shared_vpc_host_project.host[0].project + service_project = var.workspace_google_project +} diff --git a/modules/gcp/network/subnets.tf b/modules/gcp/network/subnets.tf new file mode 100644 index 00000000..c027ceff --- /dev/null +++ b/modules/gcp/network/subnets.tf @@ -0,0 +1,23 @@ +# === Spoke subnet ======================================================= +resource "google_compute_subnetwork" "spoke_subnet" { + count = local.create_spoke ? 1 : 0 + + name = local.subnet_name + project = var.spoke_vpc_google_project + network = google_compute_network.spoke_vpc[0].id + region = var.google_region + ip_cidr_range = var.subnet_cidr + private_ip_google_access = true +} + +# === Hub subnet ========================================================= +resource "google_compute_subnetwork" "hub_subnet" { + count = local.create_hub_vpc ? 1 : 0 + + name = "${var.prefix}-hub-subnet-${var.suffix}" + project = var.hub_vpc_google_project + network = google_compute_network.hub_vpc[0].id + region = var.google_region + ip_cidr_range = var.hub_vpc_cidr + private_ip_google_access = true +} diff --git a/modules/gcp/network/tests/create-shared/main.tf b/modules/gcp/network/tests/create-shared/main.tf new file mode 100644 index 00000000..a77a1f2c --- /dev/null +++ b/modules/gcp/network/tests/create-shared/main.tf @@ -0,0 +1,23 @@ +terraform { + required_version = ">= 1.5" +} + +provider "google" { + project = "fixture-project" + region = "us-central1" +} + +# Shared-VPC binding without a hub (BYOVPC + Shared VPC) +module "network" { + source = "../.." + + prefix = "fixture" + suffix = "abc123" + google_region = "us-central1" + vpc_source = "create" + spoke_vpc_google_project = "fixture-host-project" + subnet_cidr = "10.0.0.0/22" + + is_spoke_vpc_shared = true + workspace_google_project = "fixture-workspace-project" +} diff --git a/modules/gcp/network/tests/create-with-hub/main.tf b/modules/gcp/network/tests/create-with-hub/main.tf new file mode 100644 index 00000000..f1992278 --- /dev/null +++ b/modules/gcp/network/tests/create-with-hub/main.tf @@ -0,0 +1,25 @@ +terraform { + required_version = ">= 1.5" +} + +provider "google" { + project = "fixture-project" + region = "us-central1" +} + +module "network" { + source = "../.." + + prefix = "fixture" + suffix = "abc123" + google_region = "us-central1" + vpc_source = "create" + spoke_vpc_google_project = "fixture-spoke-project" + subnet_cidr = "10.0.0.0/22" + + enable_hub = true + hub_vpc_google_project = "fixture-hub-project" + hub_vpc_cidr = "10.1.0.0/24" + is_spoke_vpc_shared = true + workspace_google_project = "fixture-workspace-project" +} diff --git a/modules/gcp/network/tests/create/main.tf b/modules/gcp/network/tests/create/main.tf new file mode 100644 index 00000000..bdafcc50 --- /dev/null +++ b/modules/gcp/network/tests/create/main.tf @@ -0,0 +1,19 @@ +terraform { + required_version = ">= 1.5" +} + +provider "google" { + project = "fixture-project" + region = "us-central1" +} + +module "network" { + source = "../.." + + prefix = "fixture" + suffix = "abc123" + google_region = "us-central1" + vpc_source = "create" + spoke_vpc_google_project = "fixture-project" + subnet_cidr = "10.0.0.0/22" +} diff --git a/modules/gcp/network/tests/existing/main.tf b/modules/gcp/network/tests/existing/main.tf new file mode 100644 index 00000000..8935be2c --- /dev/null +++ b/modules/gcp/network/tests/existing/main.tf @@ -0,0 +1,20 @@ +terraform { + required_version = ">= 1.5" +} + +provider "google" { + project = "fixture-project" + region = "us-central1" +} + +module "network" { + source = "../.." + + prefix = "fixture" + suffix = "abc123" + google_region = "us-central1" + vpc_source = "existing" + spoke_vpc_google_project = "fixture-project" + existing_vpc_name = "preexisting-vpc" + existing_subnet_name = "preexisting-subnet" +} diff --git a/modules/gcp/network/variables.tf b/modules/gcp/network/variables.tf new file mode 100644 index 00000000..e8af3c32 --- /dev/null +++ b/modules/gcp/network/variables.tf @@ -0,0 +1,120 @@ +variable "prefix" { + type = string + description = "Prefix for generated resource names" +} + +variable "suffix" { + type = string + description = "Random suffix passed by the composer for uniqueness" +} + +variable "google_region" { + type = string + description = "GCP region for all network resources" +} + +variable "vpc_source" { + type = string + description = "Either 'create' (Terraform creates a VPC) or 'existing' (data-source lookup)" + validation { + condition = contains(["create", "existing"], var.vpc_source) + error_message = "vpc_source must be 'create' or 'existing'." + } +} + +# Spoke project always required +variable "spoke_vpc_google_project" { + type = string + description = "GCP project hosting the spoke VPC" +} + +# === Used when vpc_source = "create" ==================================== +variable "subnet_cidr" { + type = string + default = null + description = "CIDR for the spoke subnet (required when vpc_source=create)" +} + +variable "subnet_name" { + type = string + default = null + description = "Override for spoke subnet name (default: \"{prefix}-subnet-{suffix}\")" +} + +# === Used when vpc_source = "existing" ================================== +variable "existing_vpc_name" { + type = string + default = null + description = "Name of pre-existing VPC (required when vpc_source=existing)" +} + +variable "existing_subnet_name" { + type = string + default = null + description = "Name of pre-existing subnet (required when vpc_source=existing)" +} + +# === Hub configuration (only when enable_hub = true) ===================== +variable "enable_hub" { + type = bool + default = false + description = "Enable the hub half of the topology: hub VPC/subnet (created or looked up per hub_vpc_source) and hub-spoke peering. Composer passes restricted_egress" +} + +variable "enable_hub_spoke_peering" { + type = bool + default = true + description = "Create the bidirectional VPC peering between hub and spoke. Disable when hub-spoke connectivity is provided by other means (e.g. Shared VPC or an existing transit). Cloud DNS peering zones do not depend on it. Only takes effect when the hub is enabled" +} + +variable "hub_vpc_source" { + type = string + default = "create" + description = "Where the hub VPC comes from when the hub is enabled. create: Terraform creates the hub VPC and subnet (hub_vpc_cidr required); existing: data-source lookup of existing_hub_vpc_name/existing_hub_subnet_name in hub_vpc_google_project" + validation { + condition = contains(["create", "existing"], var.hub_vpc_source) + error_message = "hub_vpc_source must be 'create' or 'existing'." + } +} + +variable "existing_hub_vpc_name" { + type = string + default = null + description = "Name of the pre-existing hub VPC. Required when hub_vpc_source=existing" +} + +variable "existing_hub_subnet_name" { + type = string + default = null + description = "Name of the pre-existing hub subnet (must be in google_region). Required when hub_vpc_source=existing" +} + +variable "hub_vpc_google_project" { + type = string + default = null + description = "GCP project hosting the hub VPC (required when enable_hub=true)" +} + +variable "hub_vpc_cidr" { + type = string + default = null + description = "CIDR for the hub subnet (required when enable_hub=true)" +} + +variable "is_spoke_vpc_shared" { + type = bool + default = false + description = "If true, bind the spoke VPC's project as a Shared-VPC host and the workspace project as a service project" +} + +variable "workspace_google_project" { + type = string + default = null + description = "Workspace project (used for Shared-VPC service binding)" +} + +variable "enable_nat" { + type = bool + default = true + description = "Create Cloud Router + NAT for internet egress. The composer disables this under restricted_egress, where no internet egress path may exist" +} diff --git a/modules/gcp/network/versions.tf b/modules/gcp/network/versions.tf new file mode 100644 index 00000000..8b5ec227 --- /dev/null +++ b/modules/gcp/network/versions.tf @@ -0,0 +1,9 @@ +terraform { + required_version = ">= 1.5" + required_providers { + google = { + source = "hashicorp/google" + version = ">= 6.0" + } + } +} diff --git a/modules/gcp/network/vpc.tf b/modules/gcp/network/vpc.tf new file mode 100644 index 00000000..134ed2a1 --- /dev/null +++ b/modules/gcp/network/vpc.tf @@ -0,0 +1,19 @@ +# === Spoke VPC (created) ================================================ +resource "google_compute_network" "spoke_vpc" { + count = local.create_spoke ? 1 : 0 + + name = "${var.prefix}-spoke-vpc-${var.suffix}" + project = var.spoke_vpc_google_project + auto_create_subnetworks = false + routing_mode = "GLOBAL" +} + +# === Hub VPC ============================================================ +resource "google_compute_network" "hub_vpc" { + count = local.create_hub_vpc ? 1 : 0 + + name = "${var.prefix}-hub-vpc-${var.suffix}" + project = var.hub_vpc_google_project + auto_create_subnetworks = false + routing_mode = "GLOBAL" +} diff --git a/modules/gcp/private-connectivity/Makefile b/modules/gcp/private-connectivity/Makefile new file mode 100644 index 00000000..17b32ec8 --- /dev/null +++ b/modules/gcp/private-connectivity/Makefile @@ -0,0 +1,7 @@ +.PHONY: docs test_docs + +docs: + terraform-docs -c ../../../.terraform-docs.yml . + +test_docs: + terraform-docs -c ../../../.terraform-docs.yml --output-check . diff --git a/modules/gcp/private-connectivity/README.md b/modules/gcp/private-connectivity/README.md new file mode 100644 index 00000000..09c8a1f3 --- /dev/null +++ b/modules/gcp/private-connectivity/README.md @@ -0,0 +1,98 @@ +# modules/gcp/private-connectivity + +GCP-side PSC endpoints + restricted-egress firewall for the Databricks GCP composer. + +## Usage + +Typically called by `modules/gcp/databricks-workspace` (the composer). Direct consumption is supported but unusual. + +```hcl +module "private_connectivity" { + source = "github.com/databricks/terraform-databricks-examples//modules/gcp/private-connectivity" + + prefix = "acme" + suffix = "abc123" + google_region = "us-central1" + + spoke_vpc_id = module.network.spoke_vpc_id + spoke_vpc_self_link = module.network.spoke_vpc_self_link + spoke_vpc_google_project = "my-spoke-project" + spoke_vpc_cidr = "10.0.0.0/16" + + enable_frontend = true + enable_backend = true + psc_subnet_cidr = "10.0.255.0/28" +} +``` + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.5 | +| [google](#requirement\_google) | >= 6.0 | + +## Providers + +| Name | Version | +|------|---------| +| [google](#provider\_google) | 7.39.0 | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [google_compute_address.backend_address](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_address) | resource | +| [google_compute_address.frontend_address_hub](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_address) | resource | +| [google_compute_address.frontend_address_spoke](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_address) | resource | +| [google_compute_firewall.hub_ingress](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource | +| [google_compute_firewall.spoke_allow_ctl_plane](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource | +| [google_compute_firewall.spoke_allow_google_apis](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource | +| [google_compute_firewall.spoke_allow_hive](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource | +| [google_compute_firewall.spoke_default_deny_egress](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource | +| [google_compute_firewall.spoke_intra_egress](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource | +| [google_compute_firewall.spoke_intra_ingress](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_firewall) | resource | +| [google_compute_forwarding_rule.backend_forwarding_rule](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_forwarding_rule) | resource | +| [google_compute_forwarding_rule.frontend_forwarding_rule_hub](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_forwarding_rule) | resource | +| [google_compute_forwarding_rule.frontend_forwarding_rule_spoke](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_forwarding_rule) | resource | +| [google_compute_subnetwork.psc_subnet](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_subnetwork) | resource | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [google\_region](#input\_google\_region) | GCP region for PSC and firewall resources (must be one of the regions in the regional PSC service-attachment maps) | `string` | n/a | yes | +| [prefix](#input\_prefix) | Prefix used to name generated resources | `string` | n/a | yes | +| [psc\_subnet\_cidr](#input\_psc\_subnet\_cidr) | CIDR for the dedicated PSC subnet in the spoke VPC | `string` | n/a | yes | +| [spoke\_vpc\_cidr](#input\_spoke\_vpc\_cidr) | CIDR of the spoke VPC address space (used as source\_ranges for the hub ingress firewall) | `string` | n/a | yes | +| [spoke\_vpc\_google\_project](#input\_spoke\_vpc\_google\_project) | GCP project that hosts the spoke VPC | `string` | n/a | yes | +| [spoke\_vpc\_id](#input\_spoke\_vpc\_id) | ID of the spoke VPC (output from the network module) | `string` | n/a | yes | +| [spoke\_vpc\_self\_link](#input\_spoke\_vpc\_self\_link) | Self-link of the spoke VPC (used as the network reference for firewall rules) | `string` | n/a | yes | +| [suffix](#input\_suffix) | Random suffix appended to resource names for uniqueness (passed by the composer) | `string` | n/a | yes | +| [enable\_backend](#input\_enable\_backend) | Create the backend (SCC, data plane) PSC endpoint on the spoke | `bool` | `false` | no | +| [enable\_frontend](#input\_enable\_frontend) | Create the frontend (workspace UI/API) PSC endpoint on the spoke and, if hub exists, the hub side | `bool` | `false` | no | +| [enable\_hub](#input\_enable\_hub) | Whether the hub exists (composer passes restricted\_egress). Gates hub-side PSC and firewall resources; must be plan-time static | `bool` | `false` | no | +| [hive\_metastore\_ip](#input\_hive\_metastore\_ip) | Regional legacy Hive metastore IP. When set, an egress allow rule (tcp/3306) is created under restricted egress; when null, no rule is created. Workspaces using Unity Catalog (the default) do not need this. Regional IPs: https://docs.databricks.com/gcp/en/resources/ip-domain-region | `string` | `null` | no | +| [hub\_subnet\_name](#input\_hub\_subnet\_name) | Name of the hub subnet (used as the subnetwork reference for the hub-side PSC address) | `string` | `null` | no | +| [hub\_vpc\_google\_project](#input\_hub\_vpc\_google\_project) | GCP project that hosts the hub VPC (null when no hub is created) | `string` | `null` | no | +| [hub\_vpc\_id](#input\_hub\_vpc\_id) | ID of the hub VPC (null when no hub is created) | `string` | `null` | no | +| [hub\_vpc\_self\_link](#input\_hub\_vpc\_self\_link) | Self-link of the hub VPC (null when no hub is created) | `string` | `null` | no | +| [restrict\_egress](#input\_restrict\_egress) | Create the egress firewall stack: deny-egress, allow Google APIs, allow control plane, allow managed Hive (conditional), hub ingress | `bool` | `false` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| [backend\_forwarding\_rule\_name](#output\_backend\_forwarding\_rule\_name) | Name of the backend (SCC) PSC forwarding rule (null when enable\_backend=false) | +| [backend\_psc\_ip\_spoke](#output\_backend\_psc\_ip\_spoke) | IP address of the spoke-side backend PSC endpoint | +| [frontend\_forwarding\_rule\_name](#output\_frontend\_forwarding\_rule\_name) | Name of the spoke-side frontend PSC forwarding rule (null when enable\_frontend=false) | +| [frontend\_psc\_ip\_hub](#output\_frontend\_psc\_ip\_hub) | IP address of the hub-side frontend PSC endpoint (null when no hub) | +| [frontend\_psc\_ip\_spoke](#output\_frontend\_psc\_ip\_spoke) | IP address of the spoke-side frontend PSC endpoint | +| [hub\_frontend\_forwarding\_rule\_name](#output\_hub\_frontend\_forwarding\_rule\_name) | Name of the hub-side frontend PSC forwarding rule (null when no hub or no frontend) | +| [psc\_subnet\_self\_link](#output\_psc\_subnet\_self\_link) | Self-link of the PSC subnet | + diff --git a/modules/gcp/private-connectivity/firewall.tf b/modules/gcp/private-connectivity/firewall.tf new file mode 100644 index 00000000..c508eec3 --- /dev/null +++ b/modules/gcp/private-connectivity/firewall.tf @@ -0,0 +1,133 @@ +# Egress firewall stack — only emitted when restrict_egress = true. + +# === Spoke deny-egress ================================================== +resource "google_compute_firewall" "spoke_default_deny_egress" { + count = var.restrict_egress ? 1 : 0 + + name = "${var.prefix}-spoke-${var.suffix}-default-deny-egress" + project = var.spoke_vpc_google_project + network = var.spoke_vpc_self_link + + direction = "EGRESS" + priority = 1100 + destination_ranges = ["0.0.0.0/0"] + + deny { + protocol = "all" + } +} + +# === Spoke allow Google APIs ============================================ +resource "google_compute_firewall" "spoke_allow_google_apis" { + count = var.restrict_egress ? 1 : 0 + + name = "${var.prefix}-spoke-${var.suffix}-to-google-apis" + project = var.spoke_vpc_google_project + network = var.spoke_vpc_self_link + + direction = "EGRESS" + priority = 1000 + destination_ranges = [ + "199.36.153.4/30", + "199.36.153.8/30", + "34.126.0.0/18" + ] + + allow { + protocol = "all" + } +} + +# === Spoke allow Databricks control plane (to PSC IPs) ================== +resource "google_compute_firewall" "spoke_allow_ctl_plane" { + count = var.restrict_egress && var.enable_frontend && var.enable_backend ? 1 : 0 + + name = "${var.prefix}-spoke-${var.suffix}-to-databricks-control-plane" + project = var.spoke_vpc_google_project + network = var.spoke_vpc_self_link + + direction = "EGRESS" + priority = 1000 + destination_ranges = [ + "${google_compute_forwarding_rule.backend_forwarding_rule[0].ip_address}/32", + "${google_compute_forwarding_rule.frontend_forwarding_rule_spoke[0].ip_address}/32" + ] + + allow { + protocol = "tcp" + ports = ["443"] + } +} + +# === Spoke allow managed Hive (conditional on hive_metastore_ip) ======== +resource "google_compute_firewall" "spoke_allow_hive" { + count = var.restrict_egress && var.hive_metastore_ip != null ? 1 : 0 + + name = "${var.prefix}-spoke-${var.suffix}-to-${var.google_region}-managed-hive" + project = var.spoke_vpc_google_project + network = var.spoke_vpc_self_link + + direction = "EGRESS" + priority = 1000 + destination_ranges = ["${var.hive_metastore_ip}/32"] + + allow { + protocol = "tcp" + ports = ["3306"] + } +} + +# === Hub ingress from spoke ============================================= +resource "google_compute_firewall" "hub_ingress" { + count = var.restrict_egress && var.enable_hub ? 1 : 0 + + name = "${var.prefix}-hub-${var.suffix}-ingress" + project = var.hub_vpc_google_project + network = var.hub_vpc_self_link + + direction = "INGRESS" + priority = 1000 + source_ranges = [var.spoke_vpc_cidr] + + allow { + protocol = "all" + } +} + +# === Intra-VPC traffic (cluster node-to-node) =========================== +# The deny-egress rule above also covers RFC1918 space, and GCP ingress is +# implied-deny. Without these two allows, Spark clusters cannot form. +# The legacy module scoped ingress with workspace-id target_tags; this +# module runs before the workspace exists, so the rule applies VPC-wide - +# acceptable because the spoke VPC is dedicated to Databricks. +resource "google_compute_firewall" "spoke_intra_egress" { + count = var.restrict_egress ? 1 : 0 + + name = "${var.prefix}-spoke-${var.suffix}-intra-egress" + project = var.spoke_vpc_google_project + network = var.spoke_vpc_self_link + + direction = "EGRESS" + priority = 1000 + destination_ranges = [var.spoke_vpc_cidr] + + allow { + protocol = "all" + } +} + +resource "google_compute_firewall" "spoke_intra_ingress" { + count = var.restrict_egress ? 1 : 0 + + name = "${var.prefix}-spoke-${var.suffix}-intra-ingress" + project = var.spoke_vpc_google_project + network = var.spoke_vpc_self_link + + direction = "INGRESS" + priority = 1000 + source_ranges = [var.spoke_vpc_cidr] + + allow { + protocol = "all" + } +} diff --git a/modules/gcp-with-psc-exfiltration-protection/main.tf b/modules/gcp/private-connectivity/locals.tf similarity index 78% rename from modules/gcp-with-psc-exfiltration-protection/main.tf rename to modules/gcp/private-connectivity/locals.tf index c587c4b3..1bb0f02d 100644 --- a/modules/gcp-with-psc-exfiltration-protection/main.tf +++ b/modules/gcp/private-connectivity/locals.tf @@ -1,16 +1,4 @@ -##################################################### -# Local Values and Random String Resource -##################################################### - -# --------------------------------------------------- -# Local Value: Extract Workspace DNS ID -# --------------------------------------------------- locals { - # Extracts a numeric identifier from the Databricks workspace URL. - # The regex pattern "[0-9]+\.[0-9]+" matches the first occurrence of two groups of digits separated by a dot (e.g., "1234567890123456.1234"). - # This value is typically used to generate unique DNS names for the workspace. - workspace_dns_id = regex("[0-9]+\\.[0-9]+", databricks_mws_workspaces.databricks_workspace.workspace_url) - google_frontend_psc_targets = { "asia-northeast1" = "projects/general-prod-asianortheast1-01/regions/asia-northeast1/serviceAttachments/plproxy-psc-endpoint-all-ports" "asia-south1" = "projects/gen-prod-asias1-01/regions/asia-south1/serviceAttachments/plproxy-psc-endpoint-all-ports" @@ -45,19 +33,3 @@ locals { "us-west4" = "projects/prod-gcp-us-west4/regions/us-west4/serviceAttachments/ngrok-psc-endpoint" } } - -# --------------------------------------------------- -# Random String Resource: Suffix Generator -# --------------------------------------------------- -resource "random_string" "suffix" { - lifecycle { - ignore_changes = [ - special, - upper - ] - } - - special = false - upper = false - length = 6 -} diff --git a/modules/gcp/private-connectivity/outputs.tf b/modules/gcp/private-connectivity/outputs.tf new file mode 100644 index 00000000..403c3e0d --- /dev/null +++ b/modules/gcp/private-connectivity/outputs.tf @@ -0,0 +1,34 @@ +output "psc_subnet_self_link" { + value = google_compute_subnetwork.psc_subnet.self_link + description = "Self-link of the PSC subnet" +} + +output "frontend_forwarding_rule_name" { + value = var.enable_frontend ? google_compute_forwarding_rule.frontend_forwarding_rule_spoke[0].name : null + description = "Name of the spoke-side frontend PSC forwarding rule (null when enable_frontend=false)" +} + +output "backend_forwarding_rule_name" { + value = var.enable_backend ? google_compute_forwarding_rule.backend_forwarding_rule[0].name : null + description = "Name of the backend (SCC) PSC forwarding rule (null when enable_backend=false)" +} + +output "hub_frontend_forwarding_rule_name" { + value = var.enable_hub && var.enable_frontend ? google_compute_forwarding_rule.frontend_forwarding_rule_hub[0].name : null + description = "Name of the hub-side frontend PSC forwarding rule (null when no hub or no frontend)" +} + +output "frontend_psc_ip_spoke" { + value = var.enable_frontend ? google_compute_address.frontend_address_spoke[0].address : null + description = "IP address of the spoke-side frontend PSC endpoint" +} + +output "backend_psc_ip_spoke" { + value = var.enable_backend ? google_compute_address.backend_address[0].address : null + description = "IP address of the spoke-side backend PSC endpoint" +} + +output "frontend_psc_ip_hub" { + value = var.enable_hub && var.enable_frontend ? google_compute_address.frontend_address_hub[0].address : null + description = "IP address of the hub-side frontend PSC endpoint (null when no hub)" +} diff --git a/modules/gcp/private-connectivity/psc.tf b/modules/gcp/private-connectivity/psc.tf new file mode 100644 index 00000000..17800ab7 --- /dev/null +++ b/modules/gcp/private-connectivity/psc.tf @@ -0,0 +1,78 @@ +# === PSC Subnet (spoke) ================================================= +resource "google_compute_subnetwork" "psc_subnet" { + name = "${var.prefix}-psc-subnet-${var.suffix}" + project = var.spoke_vpc_google_project + network = var.spoke_vpc_id + region = var.google_region + ip_cidr_range = var.psc_subnet_cidr + private_ip_google_access = true +} + +# === Backend (SCC) PSC endpoint — spoke ================================= +resource "google_compute_address" "backend_address" { + count = var.enable_backend ? 1 : 0 + + name = "${var.prefix}-psc-scc-ip-${var.suffix}" + project = var.spoke_vpc_google_project + region = var.google_region + subnetwork = google_compute_subnetwork.psc_subnet.name + address_type = "INTERNAL" +} + +resource "google_compute_forwarding_rule" "backend_forwarding_rule" { + count = var.enable_backend ? 1 : 0 + + name = "${var.prefix}-psc-scc-ep-${var.suffix}" + project = var.spoke_vpc_google_project + region = var.google_region + network = var.spoke_vpc_id + ip_address = google_compute_address.backend_address[0].id + target = local.google_backend_psc_targets[var.google_region] + load_balancing_scheme = "" +} + +# === Frontend PSC endpoint — spoke ====================================== +resource "google_compute_address" "frontend_address_spoke" { + count = var.enable_frontend ? 1 : 0 + + name = "${var.prefix}-psc-ws-ip-${var.suffix}" + project = var.spoke_vpc_google_project + region = var.google_region + subnetwork = google_compute_subnetwork.psc_subnet.name + address_type = "INTERNAL" +} + +resource "google_compute_forwarding_rule" "frontend_forwarding_rule_spoke" { + count = var.enable_frontend ? 1 : 0 + + name = "${var.prefix}-psc-ws-ep-${var.suffix}" + project = var.spoke_vpc_google_project + region = var.google_region + network = var.spoke_vpc_id + ip_address = google_compute_address.frontend_address_spoke[0].id + target = local.google_frontend_psc_targets[var.google_region] + load_balancing_scheme = "" +} + +# === Frontend PSC endpoint — hub (transit) ============================== +resource "google_compute_address" "frontend_address_hub" { + count = var.enable_hub && var.enable_frontend ? 1 : 0 + + name = "${var.prefix}-hub-psc-ws-ip-${var.suffix}" + project = var.hub_vpc_google_project + region = var.google_region + subnetwork = var.hub_subnet_name + address_type = "INTERNAL" +} + +resource "google_compute_forwarding_rule" "frontend_forwarding_rule_hub" { + count = var.enable_hub && var.enable_frontend ? 1 : 0 + + name = "${var.prefix}-hub-psc-ws-ep-${var.suffix}" + project = var.hub_vpc_google_project + region = var.google_region + network = var.hub_vpc_id + ip_address = google_compute_address.frontend_address_hub[0].id + target = local.google_frontend_psc_targets[var.google_region] + load_balancing_scheme = "" +} diff --git a/modules/gcp/private-connectivity/tests/full-isolated/main.tf b/modules/gcp/private-connectivity/tests/full-isolated/main.tf new file mode 100644 index 00000000..229c71d3 --- /dev/null +++ b/modules/gcp/private-connectivity/tests/full-isolated/main.tf @@ -0,0 +1,32 @@ +terraform { + required_version = ">= 1.5" +} + +provider "google" { + project = "fixture-spoke" + region = "us-central1" +} + +module "pc" { + source = "../.." + + prefix = "fixture" + suffix = "abc123" + google_region = "us-central1" + + spoke_vpc_id = "projects/fixture-spoke/global/networks/spoke-vpc" + spoke_vpc_self_link = "https://www.googleapis.com/compute/v1/projects/fixture-spoke/global/networks/spoke-vpc" + spoke_vpc_google_project = "fixture-spoke" + spoke_vpc_cidr = "10.0.0.0/16" + + hub_vpc_id = "projects/fixture-hub/global/networks/hub-vpc" + hub_vpc_self_link = "https://www.googleapis.com/compute/v1/projects/fixture-hub/global/networks/hub-vpc" + hub_vpc_google_project = "fixture-hub" + hub_subnet_name = "fixture-hub-subnet-abc123" + + enable_frontend = true + enable_backend = true + restrict_egress = true + enable_hub = true + psc_subnet_cidr = "10.0.255.0/28" +} diff --git a/modules/gcp/private-connectivity/tests/no-egress/main.tf b/modules/gcp/private-connectivity/tests/no-egress/main.tf new file mode 100644 index 00000000..6c318d4c --- /dev/null +++ b/modules/gcp/private-connectivity/tests/no-egress/main.tf @@ -0,0 +1,26 @@ +terraform { + required_version = ">= 1.5" +} + +provider "google" { + project = "fixture-spoke" + region = "us-central1" +} + +module "pc" { + source = "../.." + + prefix = "fixture" + suffix = "abc123" + google_region = "us-central1" + + spoke_vpc_id = "projects/fixture-spoke/global/networks/spoke-vpc" + spoke_vpc_self_link = "https://www.googleapis.com/compute/v1/projects/fixture-spoke/global/networks/spoke-vpc" + spoke_vpc_google_project = "fixture-spoke" + spoke_vpc_cidr = "10.0.0.0/16" + + enable_frontend = true + enable_backend = false + restrict_egress = false + psc_subnet_cidr = "10.0.255.0/28" +} diff --git a/modules/gcp/private-connectivity/variables.tf b/modules/gcp/private-connectivity/variables.tf new file mode 100644 index 00000000..ce0b8b1e --- /dev/null +++ b/modules/gcp/private-connectivity/variables.tf @@ -0,0 +1,109 @@ +variable "prefix" { + type = string + description = "Prefix used to name generated resources" +} + +variable "suffix" { + type = string + description = "Random suffix appended to resource names for uniqueness (passed by the composer)" +} + +variable "google_region" { + type = string + description = "GCP region for PSC and firewall resources (must be one of the regions in the regional PSC service-attachment maps)" + validation { + condition = contains([ + "asia-northeast1", "asia-south1", "asia-southeast1", "australia-southeast1", + "europe-west1", "europe-west2", "europe-west3", "northamerica-northeast1", + "southamerica-east1", "us-central1", "us-east1", "us-east4", "us-west1", "us-west4" + ], var.google_region) + error_message = "google_region must be one of the regions in the regional PSC service-attachment maps. See locals.tf in modules/gcp/private-connectivity." + } +} + +# Spoke network refs +variable "spoke_vpc_id" { + type = string + description = "ID of the spoke VPC (output from the network module)" +} + +variable "spoke_vpc_self_link" { + type = string + description = "Self-link of the spoke VPC (used as the network reference for firewall rules)" +} + +variable "spoke_vpc_google_project" { + type = string + description = "GCP project that hosts the spoke VPC" +} + +variable "spoke_vpc_cidr" { + type = string + description = "CIDR of the spoke VPC address space (used as source_ranges for the hub ingress firewall)" +} + +# Hub network refs (nullable when no hub) +variable "hub_vpc_id" { + type = string + default = null + description = "ID of the hub VPC (null when no hub is created)" +} + +variable "hub_vpc_self_link" { + type = string + default = null + description = "Self-link of the hub VPC (null when no hub is created)" +} + +variable "hub_vpc_google_project" { + type = string + default = null + description = "GCP project that hosts the hub VPC (null when no hub is created)" +} + +variable "hub_subnet_name" { + type = string + default = null + description = "Name of the hub subnet (used as the subnetwork reference for the hub-side PSC address)" +} + +# Feature flags +variable "enable_frontend" { + type = bool + default = false + description = "Create the frontend (workspace UI/API) PSC endpoint on the spoke and, if hub exists, the hub side" +} + +variable "enable_backend" { + type = bool + default = false + description = "Create the backend (SCC, data plane) PSC endpoint on the spoke" +} + +variable "restrict_egress" { + type = bool + default = false + description = "Create the egress firewall stack: deny-egress, allow Google APIs, allow control plane, allow managed Hive (conditional), hub ingress" +} + +# PSC subnet CIDR +variable "psc_subnet_cidr" { + type = string + description = "CIDR for the dedicated PSC subnet in the spoke VPC" +} + +variable "hive_metastore_ip" { + type = string + default = null + description = "Regional legacy Hive metastore IP. When set, an egress allow rule (tcp/3306) is created under restricted egress; when null, no rule is created. Workspaces using Unity Catalog (the default) do not need this. Regional IPs: https://docs.databricks.com/gcp/en/resources/ip-domain-region" + validation { + condition = var.hive_metastore_ip == null || can(cidrnetmask("${var.hive_metastore_ip}/32")) + error_message = "hive_metastore_ip must be a valid IPv4 address, or null to skip the rule." + } +} + +variable "enable_hub" { + type = bool + default = false + description = "Whether the hub exists (composer passes restricted_egress). Gates hub-side PSC and firewall resources; must be plan-time static" +} diff --git a/modules/gcp/private-connectivity/versions.tf b/modules/gcp/private-connectivity/versions.tf new file mode 100644 index 00000000..8b5ec227 --- /dev/null +++ b/modules/gcp/private-connectivity/versions.tf @@ -0,0 +1,9 @@ +terraform { + required_version = ">= 1.5" + required_providers { + google = { + source = "hashicorp/google" + version = ">= 6.0" + } + } +} diff --git a/modules/gcp/service-account/Makefile b/modules/gcp/service-account/Makefile new file mode 100644 index 00000000..17b32ec8 --- /dev/null +++ b/modules/gcp/service-account/Makefile @@ -0,0 +1,7 @@ +.PHONY: docs test_docs + +docs: + terraform-docs -c ../../../.terraform-docs.yml . + +test_docs: + terraform-docs -c ../../../.terraform-docs.yml --output-check . diff --git a/modules/gcp-sa-provisioning/README.md b/modules/gcp/service-account/README.md similarity index 63% rename from modules/gcp-sa-provisioning/README.md rename to modules/gcp/service-account/README.md index 75b57fcf..ebe3a97f 100644 --- a/modules/gcp-sa-provisioning/README.md +++ b/modules/gcp/service-account/README.md @@ -25,16 +25,35 @@ You can do the same thing by provisioning a service account that will have the s - run `terraform init` - run `teraform apply` +## Usage + +Run once per GCP project to provision the service account Databricks uses to deploy workspaces. + +```hcl +module "service_account" { + source = "github.com/databricks/terraform-databricks-examples//modules/gcp/service-account" + + google_project = "my-project" + prefix = "acme" + delegate_from = ["user:alice@example.com"] +} +``` + +The consumer must also configure `provider "google" {}` (project + region/zone) — this module does not carry its own provider configuration. + ## Requirements -No requirements. +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.5 | +| [google](#requirement\_google) | >= 6.0 | ## Providers | Name | Version | |------|---------| -| [google](#provider\_google) | n/a | +| [google](#provider\_google) | 7.39.0 | ## Modules @@ -45,25 +64,22 @@ No modules. | Name | Type | |------|------| | [google_project_iam_custom_role.workspace_creator](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/project_iam_custom_role) | resource | -| [google_project_iam_member.sa2_can_create_workspaces](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/project_iam_member) | resource | -| [google_service_account.sa2](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account) | resource | +| [google_project_iam_member.workspace_creator](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/project_iam_member) | resource | +| [google_service_account.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account) | resource | | [google_service_account_iam_policy.impersonatable](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account_iam_policy) | resource | -| [google_client_config.current](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/client_config) | data source | -| [google_client_openid_userinfo.me](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/client_openid_userinfo) | data source | | [google_iam_policy.this](https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/iam_policy) | data source | ## Inputs | Name | Description | Type | Default | Required | |------|-------------|------|---------|:--------:| -| [delegate\_from](#input\_delegate\_from) | Identities to allow to impersonate created service account (in form of user:user.name@example.com, group:deployers@example.com or serviceAccount:sa1@project.iam.gserviceaccount.com) | `list(string)` | n/a | yes | -| [google\_project](#input\_google\_project) | Google project for VCP/workspace deployment | `string` | n/a | yes | +| [delegate\_from](#input\_delegate\_from) | Identities to allow to impersonate created service account (in form of user:user.name@example.com, group:deployers@example.com or serviceAccount:deployer@my-project.iam.gserviceaccount.com) | `list(string)` | n/a | yes | +| [google\_project](#input\_google\_project) | Google project for VPC/workspace deployment | `string` | n/a | yes | | [prefix](#input\_prefix) | Prefix to use in generated service account name | `string` | n/a | yes | ## Outputs | Name | Description | |------|-------------| -| [custom\_role\_url](#output\_custom\_role\_url) | n/a | -| [service\_account](#output\_service\_account) | Add this email as a user in the Databricks account console | +| [service\_account\_email](#output\_service\_account\_email) | Add this email as a user in the Databricks account console | diff --git a/modules/gcp/service-account/main.tf b/modules/gcp/service-account/main.tf new file mode 100644 index 00000000..c20176bd --- /dev/null +++ b/modules/gcp/service-account/main.tf @@ -0,0 +1,56 @@ +resource "google_service_account" "this" { + account_id = "${var.prefix}-ws-creator" + display_name = "Service Account for Databricks Provisioning" +} + +data "google_iam_policy" "this" { + binding { + role = "roles/iam.serviceAccountTokenCreator" + members = var.delegate_from + } +} + +resource "google_service_account_iam_policy" "impersonatable" { + service_account_id = google_service_account.this.name + policy_data = data.google_iam_policy.this.policy_data +} + +resource "google_project_iam_custom_role" "workspace_creator" { + role_id = "${var.prefix}_workspace_creator" + title = "Databricks Workspace Creator" + project = var.google_project + # Current workspace-creator permission set (managed VPC + BYOVPC union): https://docs.databricks.com/gcp/en/admin/cloud-configurations/gcp/permissions + # GKE-era container.*/storage.* permissions were removed: the data plane is GCE and + # Databricks service agents provision workspace infrastructure. + permissions = [ + "compute.firewalls.create", + "compute.firewalls.get", + "compute.firewalls.update", + "compute.networks.get", + "compute.networks.updatePolicy", + "compute.projects.get", + "compute.subnetworks.get", + "compute.subnetworks.getIamPolicy", + "compute.subnetworks.setIamPolicy", + "iam.roles.create", + "iam.roles.delete", + "iam.roles.get", + "iam.roles.update", + "iam.serviceAccounts.create", + "iam.serviceAccounts.get", + "iam.serviceAccounts.getIamPolicy", + "iam.serviceAccounts.setIamPolicy", + "resourcemanager.projects.get", + "resourcemanager.projects.getIamPolicy", + "resourcemanager.projects.setIamPolicy", + "serviceusage.services.enable", + "serviceusage.services.get", + "serviceusage.services.list", + ] +} + +resource "google_project_iam_member" "workspace_creator" { + role = google_project_iam_custom_role.workspace_creator.id + member = "serviceAccount:${google_service_account.this.email}" + project = var.google_project +} diff --git a/modules/gcp/service-account/outputs.tf b/modules/gcp/service-account/outputs.tf new file mode 100644 index 00000000..897270e4 --- /dev/null +++ b/modules/gcp/service-account/outputs.tf @@ -0,0 +1,4 @@ +output "service_account_email" { + value = google_service_account.this.email + description = "Add this email as a user in the Databricks account console" +} diff --git a/modules/gcp-sa-provisioning/variables.tf b/modules/gcp/service-account/variables.tf similarity index 71% rename from modules/gcp-sa-provisioning/variables.tf rename to modules/gcp/service-account/variables.tf index d366a50c..f4cfdbb1 100644 --- a/modules/gcp-sa-provisioning/variables.tf +++ b/modules/gcp/service-account/variables.tf @@ -1,6 +1,6 @@ variable "google_project" { type = string - description = "Google project for VCP/workspace deployment" + description = "Google project for VPC/workspace deployment" } variable "prefix" { @@ -9,6 +9,6 @@ variable "prefix" { } variable "delegate_from" { - description = "Identities to allow to impersonate created service account (in form of user:user.name@example.com, group:deployers@example.com or serviceAccount:sa1@project.iam.gserviceaccount.com)" + description = "Identities to allow to impersonate created service account (in form of user:user.name@example.com, group:deployers@example.com or serviceAccount:deployer@my-project.iam.gserviceaccount.com)" type = list(string) } diff --git a/modules/gcp/service-account/versions.tf b/modules/gcp/service-account/versions.tf new file mode 100644 index 00000000..8b5ec227 --- /dev/null +++ b/modules/gcp/service-account/versions.tf @@ -0,0 +1,9 @@ +terraform { + required_version = ">= 1.5" + required_providers { + google = { + source = "hashicorp/google" + version = ">= 6.0" + } + } +} diff --git a/modules/gcp/unity-catalog/Makefile b/modules/gcp/unity-catalog/Makefile new file mode 100644 index 00000000..17b32ec8 --- /dev/null +++ b/modules/gcp/unity-catalog/Makefile @@ -0,0 +1,7 @@ +.PHONY: docs test_docs + +docs: + terraform-docs -c ../../../.terraform-docs.yml . + +test_docs: + terraform-docs -c ../../../.terraform-docs.yml --output-check . diff --git a/modules/gcp/unity-catalog/README.md b/modules/gcp/unity-catalog/README.md new file mode 100644 index 00000000..6f12606a --- /dev/null +++ b/modules/gcp/unity-catalog/README.md @@ -0,0 +1,80 @@ +# modules/gcp/unity-catalog + +Unity Catalog metastore, GCS bucket, storage credential, external location, and catalog for GCP Databricks workspaces. Called by examples after the workspace exists (uses workspace-scoped Databricks provider alias). + +## Usage + +Called after `modules/gcp/databricks-workspace` to create a metastore, GCS bucket, storage credential, external location, and default catalog. + +```hcl +module "unity_catalog" { + source = "github.com/databricks/terraform-databricks-examples//modules/gcp/unity-catalog" + + providers = { + databricks = databricks + databricks.workspace = databricks.workspace + } + + databricks_workspace_id = module.workspace.workspace_id + databricks_workspace_url = module.workspace.workspace_url + google_project = "my-workspace-project" + google_region = "us-central1" + prefix = "acme" + metastore_name = "main-metastore" + catalog_name = "main" +} +``` + +The consumer must declare a `databricks.workspace` provider alias pointing at the workspace URL. + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.5 | +| [databricks](#requirement\_databricks) | >= 1.81.1 | +| [google](#requirement\_google) | >= 6.0 | +| [random](#requirement\_random) | >= 3.0 | + +## Providers + +| Name | Version | +|------|---------| +| [databricks](#provider\_databricks) | 1.120.0 | +| [databricks.workspace](#provider\_databricks.workspace) | 1.120.0 | +| [google](#provider\_google) | 7.39.0 | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [databricks_catalog.main](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/catalog) | resource | +| [databricks_external_location.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/external_location) | resource | +| [databricks_metastore.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/metastore) | resource | +| [databricks_metastore_assignment.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/metastore_assignment) | resource | +| [databricks_storage_credential.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/storage_credential) | resource | +| [google_storage_bucket.catalog_storage](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket) | resource | +| [google_storage_bucket_iam_member.unity_cred_admin](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket_iam_member) | resource | +| [google_storage_bucket_iam_member.unity_cred_reader](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket_iam_member) | resource | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [catalog\_name](#input\_catalog\_name) | Name to assign to default catalog | `string` | n/a | yes | +| [databricks\_workspace\_id](#input\_databricks\_workspace\_id) | The unique identifier of the Databricks workspace in which resources will be managed. | `any` | n/a | yes | +| [databricks\_workspace\_url](#input\_databricks\_workspace\_url) | The URL of the Databricks workspace to which resources will be deployed (e.g., https://.gcp.databricks.com). | `any` | n/a | yes | +| [google\_project](#input\_google\_project) | The Google Cloud project ID where the Databricks workspace and associated resources will be created. | `string` | n/a | yes | +| [google\_region](#input\_google\_region) | Google Cloud region where the resources will be created | `string` | n/a | yes | +| [metastore\_name](#input\_metastore\_name) | Name to assign to regional metastore | `string` | n/a | yes | +| [prefix](#input\_prefix) | Prefix to use in generated resources name | `string` | n/a | yes | + +## Outputs + +No outputs. + diff --git a/modules/gcp-unity-catalog/databricks-cloud-resources.tf b/modules/gcp/unity-catalog/databricks-cloud-resources.tf similarity index 94% rename from modules/gcp-unity-catalog/databricks-cloud-resources.tf rename to modules/gcp/unity-catalog/databricks-cloud-resources.tf index 003175a5..c1f85801 100644 --- a/modules/gcp-unity-catalog/databricks-cloud-resources.tf +++ b/modules/gcp/unity-catalog/databricks-cloud-resources.tf @@ -19,7 +19,7 @@ resource "databricks_storage_credential" "this" { resource "databricks_external_location" "this" { provider = databricks.workspace name = "${var.prefix}-external-location" - url = "gs://${google_storage_bucket.ext_bucket.name}/" + url = "gs://${google_storage_bucket.catalog_storage.name}/" credential_name = databricks_storage_credential.this.id diff --git a/modules/gcp-unity-catalog/gcs.tf b/modules/gcp/unity-catalog/gcs.tf similarity index 77% rename from modules/gcp-unity-catalog/gcs.tf rename to modules/gcp/unity-catalog/gcs.tf index d0982072..cb52397a 100644 --- a/modules/gcp-unity-catalog/gcs.tf +++ b/modules/gcp/unity-catalog/gcs.tf @@ -1,4 +1,4 @@ -resource "google_storage_bucket" "ext_bucket" { +resource "google_storage_bucket" "catalog_storage" { name = "${var.prefix}-bucket" project = var.google_project @@ -7,13 +7,13 @@ resource "google_storage_bucket" "ext_bucket" { } resource "google_storage_bucket_iam_member" "unity_cred_admin" { - bucket = google_storage_bucket.ext_bucket.name + bucket = google_storage_bucket.catalog_storage.name role = "roles/storage.objectAdmin" member = "serviceAccount:${databricks_storage_credential.this.databricks_gcp_service_account[0].email}" } resource "google_storage_bucket_iam_member" "unity_cred_reader" { - bucket = google_storage_bucket.ext_bucket.name + bucket = google_storage_bucket.catalog_storage.name role = "roles/storage.legacyBucketReader" member = "serviceAccount:${databricks_storage_credential.this.databricks_gcp_service_account[0].email}" } diff --git a/modules/gcp-unity-catalog/variables.tf b/modules/gcp/unity-catalog/variables.tf similarity index 100% rename from modules/gcp-unity-catalog/variables.tf rename to modules/gcp/unity-catalog/variables.tf diff --git a/modules/gcp-unity-catalog/terraform.tf b/modules/gcp/unity-catalog/versions.tf similarity index 53% rename from modules/gcp-unity-catalog/terraform.tf rename to modules/gcp/unity-catalog/versions.tf index af49a607..4e23535f 100644 --- a/modules/gcp-unity-catalog/terraform.tf +++ b/modules/gcp/unity-catalog/versions.tf @@ -1,14 +1,18 @@ terraform { + required_version = ">= 1.5" required_providers { databricks = { source = "databricks/databricks" configuration_aliases = [databricks, databricks.workspace] + version = ">= 1.81.1" } google = { - source = "hashicorp/google" + source = "hashicorp/google" + version = ">= 6.0" } random = { - source = "hashicorp/random" + source = "hashicorp/random" + version = ">= 3.0" } } -} \ No newline at end of file +} diff --git a/modules/gcp/workspace/Makefile b/modules/gcp/workspace/Makefile new file mode 100644 index 00000000..17b32ec8 --- /dev/null +++ b/modules/gcp/workspace/Makefile @@ -0,0 +1,7 @@ +.PHONY: docs test_docs + +docs: + terraform-docs -c ../../../.terraform-docs.yml . + +test_docs: + terraform-docs -c ../../../.terraform-docs.yml --output-check . diff --git a/modules/gcp/workspace/README.md b/modules/gcp/workspace/README.md new file mode 100644 index 00000000..81c83f46 --- /dev/null +++ b/modules/gcp/workspace/README.md @@ -0,0 +1,99 @@ +# modules/gcp/workspace + +Owns workspace registration with the Databricks control plane: `databricks_mws_*` resources for the GCP composer, including `mws_networks`, `mws_workspaces`, `mws_vpc_endpoint`, `mws_private_access_settings`. + +## Usage + +Typically called by `modules/gcp/databricks-workspace` (the composer). Direct consumption is supported but unusual. + +```hcl +module "workspace" { + source = "github.com/databricks/terraform-databricks-examples//modules/gcp/workspace" + + prefix = "acme" + suffix = "abc123" + databricks_account_id = var.databricks_account_id + google_project = "my-workspace-project" + google_region = "us-central1" + vpc_source = "databricks_managed" +} +``` + + +## Requirements + +| Name | Version | +|------|---------| +| [terraform](#requirement\_terraform) | >= 1.5 | +| [databricks](#requirement\_databricks) | >= 1.81.1 | + +## Providers + +| Name | Version | +|------|---------| +| [databricks](#provider\_databricks) | 1.120.0 | +| [terraform](#provider\_terraform) | n/a | + +## Modules + +No modules. + +## Resources + +| Name | Type | +|------|------| +| [databricks_account_network_policy.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/account_network_policy) | resource | +| [databricks_mws_customer_managed_keys.managed_services](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_customer_managed_keys) | resource | +| [databricks_mws_customer_managed_keys.storage](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_customer_managed_keys) | resource | +| [databricks_mws_networks.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_networks) | resource | +| [databricks_mws_private_access_settings.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_private_access_settings) | resource | +| [databricks_mws_vpc_endpoint.backend](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_vpc_endpoint) | resource | +| [databricks_mws_vpc_endpoint.frontend](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_vpc_endpoint) | resource | +| [databricks_mws_vpc_endpoint.transit](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_vpc_endpoint) | resource | +| [databricks_mws_workspaces.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/mws_workspaces) | resource | +| [databricks_workspace_network_option.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/workspace_network_option) | resource | +| [terraform_data.nat_gate](https://registry.terraform.io/providers/hashicorp/terraform/latest/docs/resources/data) | resource | + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| [databricks\_account\_id](#input\_databricks\_account\_id) | Databricks account ID (GUID) where this workspace will be registered | `string` | n/a | yes | +| [google\_project](#input\_google\_project) | GCP project ID hosting the workspace data plane | `string` | n/a | yes | +| [google\_region](#input\_google\_region) | GCP region where the workspace will be deployed | `string` | n/a | yes | +| [prefix](#input\_prefix) | Prefix used to name generated resources | `string` | n/a | yes | +| [suffix](#input\_suffix) | Random suffix appended to resource names for uniqueness (passed by the composer) | `string` | n/a | yes | +| [vpc\_source](#input\_vpc\_source) | One of: databricks\_managed (no mws\_networks), create (we built the VPC), existing (data-source lookup) | `string` | n/a | yes | +| [backend\_forwarding\_rule\_name](#input\_backend\_forwarding\_rule\_name) | Name of the backend (SCC) PSC forwarding rule from private-connectivity; used as gcp\_vpc\_endpoint\_info.psc\_endpoint\_name | `string` | `null` | no | +| [cmek\_managed\_services\_key\_id](#input\_cmek\_managed\_services\_key\_id) | Cloud KMS key resource ID for managed-services CMEK (control-plane data: notebooks, secrets, queries). Null disables. The principal running Terraform needs cloudkms.cryptoKeys.getIamPolicy and setIamPolicy on the key - Databricks sets the key's IAM policy at workspace creation. Enterprise tier; set at creation only. The key must exist before plan (a key created in the same configuration makes the count unknown and fails plan) | `string` | `null` | no | +| [cmek\_storage\_key\_id](#input\_cmek\_storage\_key\_id) | Cloud KMS key resource ID for workspace-storage CMEK (GCS buckets and GCE persistent disks). Null disables. Same permission and tier requirements as cmek\_managed\_services\_key\_id; set at creation only. The key must exist before plan (a key created in the same configuration makes the count unknown and fails plan) | `string` | `null` | no | +| [enable\_backend](#input\_enable\_backend) | Create the backend (SCC) mws\_vpc\_endpoint | `bool` | `false` | no | +| [enable\_frontend](#input\_enable\_frontend) | Create the frontend mws\_vpc\_endpoint (and, if hub\_frontend\_forwarding\_rule\_name is set, the transit endpoint) | `bool` | `false` | no | +| [enable\_hub](#input\_enable\_hub) | Whether the hub exists (composer passes restricted\_egress). Gates the transit mws\_vpc\_endpoint; must be plan-time static | `bool` | `false` | no | +| [frontend\_forwarding\_rule\_name](#input\_frontend\_forwarding\_rule\_name) | Name of the frontend PSC forwarding rule from private-connectivity; used as gcp\_vpc\_endpoint\_info.psc\_endpoint\_name | `string` | `null` | no | +| [hub\_frontend\_forwarding\_rule\_name](#input\_hub\_frontend\_forwarding\_rule\_name) | Name of the hub-side frontend PSC forwarding rule from private-connectivity; used as gcp\_vpc\_endpoint\_info.psc\_endpoint\_name | `string` | `null` | no | +| [hub\_vpc\_google\_project](#input\_hub\_vpc\_google\_project) | GCP project hosting the hub VPC (used for the transit databricks\_mws\_vpc\_endpoint when restricted\_egress is enabled) | `string` | `null` | no | +| [nat\_dependency](#input\_nat\_dependency) | Opaque value (typically the Cloud NAT ID) used as depends\_on for the workspace to ensure NAT readiness before workspace creation | `any` | `null` | no | +| [private\_access\_only](#input\_private\_access\_only) | Create databricks\_mws\_private\_access\_settings with public\_access\_enabled=false and attach it to the workspace | `bool` | `false` | no | +| [serverless\_allowed\_internet\_destinations](#input\_serverless\_allowed\_internet\_destinations) | FQDNs serverless workloads may reach when serverless\_egress\_mode=restricted (max 100) | `list(string)` | `[]` | no | +| [serverless\_allowed\_storage\_destinations](#input\_serverless\_allowed\_storage\_destinations) | GCS bucket names serverless workloads may reach when serverless\_egress\_mode=restricted (max 100); region is taken from google\_region | `list(string)` | `[]` | no | +| [serverless\_egress\_enforcement](#input\_serverless\_egress\_enforcement) | enforced: violations are blocked; dry\_run: violations are only logged (use to evaluate a policy before enforcing) | `string` | `"enforced"` | no | +| [serverless\_egress\_mode](#input\_serverless\_egress\_mode) | Serverless egress control. unmanaged: no network policy resources; full: policy with FULL\_ACCESS; restricted: deny-by-default policy allowing only the listed destinations. Requires the workspace to be on the Enterprise tier | `string` | `"unmanaged"` | no | +| [spoke\_subnet\_name](#input\_spoke\_subnet\_name) | Name of the spoke subnet used in databricks\_mws\_networks.gcp\_network\_info.subnet\_id (null when vpc\_source=databricks\_managed) | `string` | `null` | no | +| [spoke\_vpc\_google\_project](#input\_spoke\_vpc\_google\_project) | GCP project hosting the spoke VPC (used in databricks\_mws\_networks.gcp\_network\_info.network\_project\_id) | `string` | `null` | no | +| [spoke\_vpc\_name](#input\_spoke\_vpc\_name) | Name of the spoke VPC used in databricks\_mws\_networks.gcp\_network\_info.vpc\_id (null when vpc\_source=databricks\_managed) | `string` | `null` | no | +| [workspace\_name](#input\_workspace\_name) | Optional workspace name override. Defaults to "prefix-ws-suffix" when null | `string` | `null` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| [backend\_endpoint\_id](#output\_backend\_endpoint\_id) | Backend mws\_vpc\_endpoint ID (null when no PSC) | +| [frontend\_endpoint\_id](#output\_frontend\_endpoint\_id) | Frontend mws\_vpc\_endpoint ID (null when no PSC) | +| [network\_id](#output\_network\_id) | mws\_networks ID (null when databricks\_managed) | +| [private\_access\_settings\_id](#output\_private\_access\_settings\_id) | databricks\_mws\_private\_access\_settings ID (null when private\_access\_only=false) | +| [serverless\_network\_policy\_id](#output\_serverless\_network\_policy\_id) | Serverless egress network-policy ID bound to the workspace (null when serverless\_egress\_mode=unmanaged) | +| [transit\_endpoint\_id](#output\_transit\_endpoint\_id) | Hub-side mws\_vpc\_endpoint ID (null when no hub) | +| [workspace\_id](#output\_workspace\_id) | Databricks workspace ID | +| [workspace\_url](#output\_workspace\_url) | Databricks workspace URL | + diff --git a/modules/gcp/workspace/cmek.tf b/modules/gcp/workspace/cmek.tf new file mode 100644 index 00000000..8afda31d --- /dev/null +++ b/modules/gcp/workspace/cmek.tf @@ -0,0 +1,19 @@ +resource "databricks_mws_customer_managed_keys" "managed_services" { + count = var.cmek_managed_services_key_id != null ? 1 : 0 + + account_id = var.databricks_account_id + gcp_key_info { + kms_key_id = var.cmek_managed_services_key_id + } + use_cases = ["MANAGED_SERVICES"] +} + +resource "databricks_mws_customer_managed_keys" "storage" { + count = var.cmek_storage_key_id != null ? 1 : 0 + + account_id = var.databricks_account_id + gcp_key_info { + kms_key_id = var.cmek_storage_key_id + } + use_cases = ["STORAGE"] +} diff --git a/modules/gcp/workspace/locals.tf b/modules/gcp/workspace/locals.tf new file mode 100644 index 00000000..b9538d0b --- /dev/null +++ b/modules/gcp/workspace/locals.tf @@ -0,0 +1,9 @@ +locals { + workspace_name = coalesce(var.workspace_name, "${var.prefix}-ws-${var.suffix}") + emit_mws_networks = var.vpc_source != "databricks_managed" + emit_vpc_endpoints = var.enable_frontend && var.enable_backend + emit_pas = var.private_access_only + + manage_serverless_egress = var.serverless_egress_mode != "unmanaged" + serverless_restricted = var.serverless_egress_mode == "restricted" +} diff --git a/modules/gcp/workspace/networks.tf b/modules/gcp/workspace/networks.tf new file mode 100644 index 00000000..e044bad7 --- /dev/null +++ b/modules/gcp/workspace/networks.tf @@ -0,0 +1,21 @@ +resource "databricks_mws_networks" "this" { + count = local.emit_mws_networks ? 1 : 0 + + account_id = var.databricks_account_id + network_name = "${var.prefix}-ntw-${var.suffix}" + + gcp_network_info { + network_project_id = var.spoke_vpc_google_project + vpc_id = var.spoke_vpc_name + subnet_id = var.spoke_subnet_name + subnet_region = var.google_region + } + + dynamic "vpc_endpoints" { + for_each = local.emit_vpc_endpoints ? [1] : [] + content { + dataplane_relay = [databricks_mws_vpc_endpoint.backend[0].vpc_endpoint_id] + rest_api = [databricks_mws_vpc_endpoint.frontend[0].vpc_endpoint_id] + } + } +} diff --git a/modules/gcp/workspace/outputs.tf b/modules/gcp/workspace/outputs.tf new file mode 100644 index 00000000..8469db4f --- /dev/null +++ b/modules/gcp/workspace/outputs.tf @@ -0,0 +1,39 @@ +output "workspace_id" { + value = databricks_mws_workspaces.this.workspace_id + description = "Databricks workspace ID" +} + +output "workspace_url" { + value = databricks_mws_workspaces.this.workspace_url + description = "Databricks workspace URL" +} + +output "network_id" { + value = local.emit_mws_networks ? databricks_mws_networks.this[0].network_id : null + description = "mws_networks ID (null when databricks_managed)" +} + +output "frontend_endpoint_id" { + value = var.enable_frontend ? databricks_mws_vpc_endpoint.frontend[0].vpc_endpoint_id : null + description = "Frontend mws_vpc_endpoint ID (null when no PSC)" +} + +output "backend_endpoint_id" { + value = var.enable_backend ? databricks_mws_vpc_endpoint.backend[0].vpc_endpoint_id : null + description = "Backend mws_vpc_endpoint ID (null when no PSC)" +} + +output "transit_endpoint_id" { + value = var.enable_frontend && var.enable_hub ? databricks_mws_vpc_endpoint.transit[0].vpc_endpoint_id : null + description = "Hub-side mws_vpc_endpoint ID (null when no hub)" +} + +output "private_access_settings_id" { + value = local.emit_pas ? databricks_mws_private_access_settings.this[0].private_access_settings_id : null + description = "databricks_mws_private_access_settings ID (null when private_access_only=false)" +} + +output "serverless_network_policy_id" { + value = local.manage_serverless_egress ? databricks_account_network_policy.this[0].network_policy_id : null + description = "Serverless egress network-policy ID bound to the workspace (null when serverless_egress_mode=unmanaged)" +} diff --git a/modules/gcp/workspace/pas.tf b/modules/gcp/workspace/pas.tf new file mode 100644 index 00000000..8c5e0655 --- /dev/null +++ b/modules/gcp/workspace/pas.tf @@ -0,0 +1,8 @@ +resource "databricks_mws_private_access_settings" "this" { + count = local.emit_pas ? 1 : 0 + + private_access_settings_name = "${var.prefix}-pas-${var.suffix}" + region = var.google_region + public_access_enabled = false + private_access_level = "ACCOUNT" +} diff --git a/modules/gcp/workspace/serverless-egress.tf b/modules/gcp/workspace/serverless-egress.tf new file mode 100644 index 00000000..f1422581 --- /dev/null +++ b/modules/gcp/workspace/serverless-egress.tf @@ -0,0 +1,37 @@ +resource "databricks_account_network_policy" "this" { + count = local.manage_serverless_egress ? 1 : 0 + + network_policy_id = "${var.prefix}-serverless-egress-${var.suffix}" + + egress = { + network_access = { + restriction_mode = local.serverless_restricted ? "RESTRICTED_ACCESS" : "FULL_ACCESS" + + allowed_internet_destinations = local.serverless_restricted ? [ + for d in var.serverless_allowed_internet_destinations : { + destination = d + internet_destination_type = "DNS_NAME" + } + ] : null + + allowed_storage_destinations = local.serverless_restricted ? [ + for b in var.serverless_allowed_storage_destinations : { + bucket_name = b + region = var.google_region + storage_destination_type = "GOOGLE_CLOUD_STORAGE" + } + ] : null + + policy_enforcement = { + enforcement_mode = var.serverless_egress_enforcement == "dry_run" ? "DRY_RUN" : "ENFORCED" + } + } + } +} + +resource "databricks_workspace_network_option" "this" { + count = local.manage_serverless_egress ? 1 : 0 + + workspace_id = databricks_mws_workspaces.this.workspace_id + network_policy_id = databricks_account_network_policy.this[0].network_policy_id +} diff --git a/modules/gcp/workspace/tests/byovpc/main.tf b/modules/gcp/workspace/tests/byovpc/main.tf new file mode 100644 index 00000000..5db9b36e --- /dev/null +++ b/modules/gcp/workspace/tests/byovpc/main.tf @@ -0,0 +1,27 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { + source = "databricks/databricks" + } + } +} + +provider "databricks" { + host = "https://accounts.gcp.databricks.com" + account_id = "00000000-0000-0000-0000-000000000000" +} + +module "workspace" { + source = "../.." + + prefix = "fixture" + suffix = "abc123" + databricks_account_id = "00000000-0000-0000-0000-000000000000" + google_project = "fixture-workspace" + google_region = "us-central1" + vpc_source = "create" + spoke_vpc_name = "fixture-spoke-vpc-abc123" + spoke_subnet_name = "fixture-subnet-abc123" + spoke_vpc_google_project = "fixture-spoke" +} diff --git a/modules/gcp/workspace/tests/databricks-managed/main.tf b/modules/gcp/workspace/tests/databricks-managed/main.tf new file mode 100644 index 00000000..8152a727 --- /dev/null +++ b/modules/gcp/workspace/tests/databricks-managed/main.tf @@ -0,0 +1,24 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { + source = "databricks/databricks" + } + } +} + +provider "databricks" { + host = "https://accounts.gcp.databricks.com" + account_id = "00000000-0000-0000-0000-000000000000" +} + +module "workspace" { + source = "../.." + + prefix = "fixture" + suffix = "abc123" + databricks_account_id = "00000000-0000-0000-0000-000000000000" + google_project = "fixture-workspace" + google_region = "us-central1" + vpc_source = "databricks_managed" +} diff --git a/modules/gcp/workspace/tests/psc-with-pas/main.tf b/modules/gcp/workspace/tests/psc-with-pas/main.tf new file mode 100644 index 00000000..5cd9a0a5 --- /dev/null +++ b/modules/gcp/workspace/tests/psc-with-pas/main.tf @@ -0,0 +1,41 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { + source = "databricks/databricks" + } + } +} + +provider "databricks" { + host = "https://accounts.gcp.databricks.com" + account_id = "00000000-0000-0000-0000-000000000000" +} + +module "workspace" { + source = "../.." + + prefix = "fixture" + suffix = "abc123" + databricks_account_id = "00000000-0000-0000-0000-000000000000" + google_project = "fixture-workspace" + google_region = "us-central1" + vpc_source = "create" + spoke_vpc_name = "fixture-spoke-vpc-abc123" + spoke_subnet_name = "fixture-subnet-abc123" + spoke_vpc_google_project = "fixture-spoke" + hub_vpc_google_project = "fixture-hub" + + frontend_forwarding_rule_name = "fixture-psc-ws-ep-abc123" + backend_forwarding_rule_name = "fixture-psc-scc-ep-abc123" + hub_frontend_forwarding_rule_name = "fixture-hub-psc-ws-ep-abc123" + + enable_frontend = true + enable_backend = true + private_access_only = true + enable_hub = true + + serverless_egress_mode = "restricted" + serverless_allowed_internet_destinations = ["pypi.org"] + serverless_allowed_storage_destinations = ["fixture-allowed-bucket"] +} diff --git a/modules/gcp/workspace/variables.tf b/modules/gcp/workspace/variables.tf new file mode 100644 index 00000000..2511b221 --- /dev/null +++ b/modules/gcp/workspace/variables.tf @@ -0,0 +1,157 @@ +variable "prefix" { + type = string + description = "Prefix used to name generated resources" +} + +variable "suffix" { + type = string + description = "Random suffix appended to resource names for uniqueness (passed by the composer)" +} + +variable "workspace_name" { + type = string + default = null + description = "Optional workspace name override. Defaults to \"prefix-ws-suffix\" when null" +} + +variable "databricks_account_id" { + type = string + description = "Databricks account ID (GUID) where this workspace will be registered" +} + +variable "google_project" { + type = string + description = "GCP project ID hosting the workspace data plane" +} + +variable "google_region" { + type = string + description = "GCP region where the workspace will be deployed" +} + +variable "vpc_source" { + type = string + description = "One of: databricks_managed (no mws_networks), create (we built the VPC), existing (data-source lookup)" + validation { + condition = contains(["databricks_managed", "create", "existing"], var.vpc_source) + error_message = "vpc_source must be one of: databricks_managed, create, existing." + } +} + +variable "spoke_vpc_name" { + type = string + default = null + description = "Name of the spoke VPC used in databricks_mws_networks.gcp_network_info.vpc_id (null when vpc_source=databricks_managed)" +} + +variable "spoke_subnet_name" { + type = string + default = null + description = "Name of the spoke subnet used in databricks_mws_networks.gcp_network_info.subnet_id (null when vpc_source=databricks_managed)" +} + +variable "spoke_vpc_google_project" { + type = string + default = null + description = "GCP project hosting the spoke VPC (used in databricks_mws_networks.gcp_network_info.network_project_id)" +} + +variable "hub_vpc_google_project" { + type = string + default = null + description = "GCP project hosting the hub VPC (used for the transit databricks_mws_vpc_endpoint when restricted_egress is enabled)" +} + +# Forwarding-rule names from private-connectivity module (gate vpc_endpoint creation) +variable "frontend_forwarding_rule_name" { + type = string + default = null + description = "Name of the frontend PSC forwarding rule from private-connectivity; used as gcp_vpc_endpoint_info.psc_endpoint_name" +} + +variable "backend_forwarding_rule_name" { + type = string + default = null + description = "Name of the backend (SCC) PSC forwarding rule from private-connectivity; used as gcp_vpc_endpoint_info.psc_endpoint_name" +} + +variable "hub_frontend_forwarding_rule_name" { + type = string + default = null + description = "Name of the hub-side frontend PSC forwarding rule from private-connectivity; used as gcp_vpc_endpoint_info.psc_endpoint_name" +} + +variable "enable_frontend" { + type = bool + default = false + description = "Create the frontend mws_vpc_endpoint (and, if hub_frontend_forwarding_rule_name is set, the transit endpoint)" +} + +variable "enable_backend" { + type = bool + default = false + description = "Create the backend (SCC) mws_vpc_endpoint" +} + +variable "private_access_only" { + type = bool + default = false + description = "Create databricks_mws_private_access_settings with public_access_enabled=false and attach it to the workspace" +} + +variable "nat_dependency" { + type = any + default = null + description = "Opaque value (typically the Cloud NAT ID) used as depends_on for the workspace to ensure NAT readiness before workspace creation" +} + +variable "enable_hub" { + type = bool + default = false + description = "Whether the hub exists (composer passes restricted_egress). Gates the transit mws_vpc_endpoint; must be plan-time static" +} + +variable "serverless_egress_mode" { + type = string + default = "unmanaged" + description = "Serverless egress control. unmanaged: no network policy resources; full: policy with FULL_ACCESS; restricted: deny-by-default policy allowing only the listed destinations. Requires the workspace to be on the Enterprise tier" + validation { + condition = contains(["unmanaged", "full", "restricted"], var.serverless_egress_mode) + error_message = "serverless_egress_mode must be one of: unmanaged, full, restricted." + } +} + +variable "serverless_allowed_internet_destinations" { + type = list(string) + default = [] + description = "FQDNs serverless workloads may reach when serverless_egress_mode=restricted (max 100)" +} + +variable "serverless_allowed_storage_destinations" { + type = list(string) + default = [] + description = "GCS bucket names serverless workloads may reach when serverless_egress_mode=restricted (max 100); region is taken from google_region" +} + +variable "serverless_egress_enforcement" { + type = string + default = "enforced" + description = "enforced: violations are blocked; dry_run: violations are only logged (use to evaluate a policy before enforcing)" + validation { + condition = contains(["enforced", "dry_run"], var.serverless_egress_enforcement) + error_message = "serverless_egress_enforcement must be one of: enforced, dry_run." + } +} + +# === Customer-managed keys (CMEK) ======================================= +variable "cmek_managed_services_key_id" { + type = string + default = null + description = "Cloud KMS key resource ID for managed-services CMEK (control-plane data: notebooks, secrets, queries). Null disables. The principal running Terraform needs cloudkms.cryptoKeys.getIamPolicy and setIamPolicy on the key - Databricks sets the key's IAM policy at workspace creation. Enterprise tier; set at creation only. The key must exist before plan (a key created in the same configuration makes the count unknown and fails plan)" +} + +variable "cmek_storage_key_id" { + type = string + default = null + description = "Cloud KMS key resource ID for workspace-storage CMEK (GCS buckets and GCE persistent disks). Null disables. Same permission and tier requirements as cmek_managed_services_key_id; set at creation only. The key must exist before plan (a key created in the same configuration makes the count unknown and fails plan)" +} diff --git a/modules/gcp/workspace/versions.tf b/modules/gcp/workspace/versions.tf new file mode 100644 index 00000000..11aab3e6 --- /dev/null +++ b/modules/gcp/workspace/versions.tf @@ -0,0 +1,9 @@ +terraform { + required_version = ">= 1.5" + required_providers { + databricks = { + source = "databricks/databricks" + version = ">= 1.81.1" + } + } +} diff --git a/modules/gcp/workspace/vpc-endpoints.tf b/modules/gcp/workspace/vpc-endpoints.tf new file mode 100644 index 00000000..0b4bf265 --- /dev/null +++ b/modules/gcp/workspace/vpc-endpoints.tf @@ -0,0 +1,41 @@ +resource "databricks_mws_vpc_endpoint" "frontend" { + count = var.enable_frontend ? 1 : 0 + + # account_id is required at runtime: the provider builds this resource's API + # path from the attribute (/accounts//vpc-endpoints), not from + # the provider config. Omitting it fails apply with a misleading OAuth error. + account_id = var.databricks_account_id + vpc_endpoint_name = "${var.prefix}-ws-ep-${var.suffix}" + + gcp_vpc_endpoint_info { + project_id = var.spoke_vpc_google_project + psc_endpoint_name = var.frontend_forwarding_rule_name + endpoint_region = var.google_region + } +} + +resource "databricks_mws_vpc_endpoint" "backend" { + count = var.enable_backend ? 1 : 0 + + account_id = var.databricks_account_id + vpc_endpoint_name = "${var.prefix}-scc-ep-${var.suffix}" + + gcp_vpc_endpoint_info { + project_id = var.spoke_vpc_google_project + psc_endpoint_name = var.backend_forwarding_rule_name + endpoint_region = var.google_region + } +} + +resource "databricks_mws_vpc_endpoint" "transit" { + count = var.enable_frontend && var.enable_hub ? 1 : 0 + + account_id = var.databricks_account_id + vpc_endpoint_name = "${var.prefix}-hub-ep-${var.suffix}" + + gcp_vpc_endpoint_info { + project_id = var.hub_vpc_google_project + psc_endpoint_name = var.hub_frontend_forwarding_rule_name + endpoint_region = var.google_region + } +} diff --git a/modules/gcp/workspace/workspace.tf b/modules/gcp/workspace/workspace.tf new file mode 100644 index 00000000..eee465e9 --- /dev/null +++ b/modules/gcp/workspace/workspace.tf @@ -0,0 +1,24 @@ +# Bridge an opaque upstream dependency (Cloud NAT readiness) into the graph. +resource "terraform_data" "nat_gate" { + input = var.nat_dependency +} + +resource "databricks_mws_workspaces" "this" { + account_id = var.databricks_account_id + workspace_name = local.workspace_name + location = var.google_region + + cloud_resource_container { + gcp { + project_id = var.google_project + } + } + + network_id = local.emit_mws_networks ? databricks_mws_networks.this[0].network_id : null + private_access_settings_id = local.emit_pas ? databricks_mws_private_access_settings.this[0].private_access_settings_id : null + + managed_services_customer_managed_key_id = var.cmek_managed_services_key_id != null ? databricks_mws_customer_managed_keys.managed_services[0].customer_managed_key_id : null + storage_customer_managed_key_id = var.cmek_storage_key_id != null ? databricks_mws_customer_managed_keys.storage[0].customer_managed_key_id : null + + depends_on = [terraform_data.nat_gate] +}