feat: Add Azure UC external location file-events module

README.md

@@ -51,6 +51,7 @@ The folder `examples` contains the following Terraform implementation examples :
 | Azure | [adb-uc](examples/adb-uc/)                                                         | ADB Unity Catalog Process                                                                                                                                                                            |
 | Azure | [adb-unity-catalog-basic-demo](examples/adb-unity-catalog-basic-demo/)             | ADB Unity Catalog end-to-end demo including UC metastore setup, Users/groups sync from AAD to databricks account, UC Catalog, External locations, Schemas, & Access Grants                           |
 | Azure | [adb-overwatch](examples/adb-overwatch/)             | Overwatch multi-workspace deployment on Azure                          |
+| Azure | [adb-uc-external-location-file-events](examples/adb-uc-external-location-file-events/) | UC external location with managed AQS file events, documented Azure RBAC, and grants |
 | AWS   | [aws-workspace-basic](examples/aws-workspace-basic/)                               | Provisioning AWS Databricks E2                                                                                                                                                                       |
 | AWS   | [aws-workspace-with-firewall](examples/aws-workspace-with-firewall/)               | Provisioning AWS Databricks E2 with an AWS Firewall                                                                                                                                                  |
 | AWS   | [aws-exfiltration-protection](examples/aws-exfiltration-protection/)               | An implementation of [Data Exfiltration Protection on AWS](https://www.databricks.com/blog/2021/02/02/data-exfiltration-protection-with-databricks-on-aws.html)                                      |
@@ -82,6 +83,7 @@ The folder `modules` contains the following Terraform modules :
 | Azure | [adb-overwatch-main-ws](modules/adb-overwatch-main-ws/)                                                   | Main Overwatch workspace deployment                                                                                                                                                       |
 | Azure | [adb-overwatch-ws-to-monitor](modules/adb-overwatch-ws-to-monitor/)                                       | Overwatch deployment on the Azure workspace to monitor                                                                                                                                    |
 | Azure | [adb-overwatch-analysis](modules/adb-overwatch-analysis/)                                                 | Overwatch analysis notebooks deployment on Azure                                                                                                                                          |
+| Azure | [adb-uc-external-location-file-events](modules/adb-uc-external-location-file-events/)                     | UC storage credential + external locations with managed AQS file events and documented Azure RBAC                                                                                         |
 | AWS   | [aws-workspace-basic](modules/aws-workspace-basic/)                                                       | Provisioning AWS Databricks E2                                                                                                                                                            |
 | AWS   | [aws-databricks-base-infra](modules/aws-databricks-base-infra/)                                           | Provisioning AWS Infrastructure to be used for the deployment of a Databricks E2 workspace                                                                                                |
 | AWS   | [aws-databricks-unity-catalog](modules/aws-databricks-unity-catalog/)                                     | Provisioning the AWS Infrastructure and setting up the metastore for Databricks Unity Catalog                                                                                             |

examples/adb-uc-external-location-file-events/README.md

@@ -0,0 +1,29 @@
+# Azure UC external location with managed file events
+
+Provisions a Unity Catalog storage credential and external location with
+**automatic managed AQS file events**, plus the documented Azure RBAC roles and
+optional UC grants.
+
+See the [module README](../../modules/adb-uc-external-location-file-events/README.md)
+for RBAC details and [file arrival triggers](https://docs.databricks.com/aws/en/jobs/file-arrival-triggers).
+
+## Prerequisites
+
+- UC-enabled Azure Databricks workspace
+- Existing ADLS Gen2 storage account and Access Connector
+- `az login` (or Azure env credentials) with permission to assign RBAC
+- Databricks CLI auth to the workspace (`databricks auth login --host ...`)
+
+## Usage
+
+1. Copy `terraform.tfvars.example` to `terraform.tfvars` and fill in values.
+2. `terraform init`
+3. `terraform plan`
+4. `terraform apply`
+
+## What gets created
+
+- Storage credential (Azure managed identity via access connector)
+- External location with `enable_file_events = true` and `file_event_queue.managed_aqs`
+- Azure RBAC: Blob Data Contributor/Reader, Queue Data Contributor, Storage Account Contributor, EventGrid EventSubscription Contributor
+- Optional UC grants on credential and location

examples/adb-uc-external-location-file-events/main.tf

@@ -0,0 +1,30 @@
+module "uc_external_location_file_events" {
+  source = "../../modules/adb-uc-external-location-file-events"
+
+  name_prefix         = var.name_prefix
+  access_connector_id = var.access_connector_id
+  storage_account_id  = var.storage_account_id
+  resource_group_name = var.resource_group_name
+
+  external_locations = [
+    {
+      name    = var.external_location_name
+      url     = var.external_location_url
+      comment = "Landing zone with managed AQS file events"
+    }
+  ]
+
+  credential_grants = var.grant_principal == "" ? [] : [
+    {
+      principal  = var.grant_principal
+      privileges = ["CREATE_EXTERNAL_LOCATION", "READ_FILES", "WRITE_FILES"]
+    }
+  ]
+
+  location_grants = var.grant_principal == "" ? [] : [
+    {
+      principal  = var.grant_principal
+      privileges = ["BROWSE", "READ_FILES", "WRITE_FILES", "CREATE_EXTERNAL_TABLE", "CREATE_EXTERNAL_VOLUME"]
+    }
+  ]
+}

examples/adb-uc-external-location-file-events/outputs.tf

@@ -0,0 +1,11 @@
+output "storage_credential_name" {
+  value = module.uc_external_location_file_events.storage_credential_name
+}
+
+output "external_location_ids" {
+  value = module.uc_external_location_file_events.external_location_ids
+}
+
+output "azure_rbac_roles" {
+  value = module.uc_external_location_file_events.azure_rbac_roles
+}

examples/adb-uc-external-location-file-events/providers.tf

@@ -0,0 +1,10 @@
+provider "azurerm" {
+  features {}
+}
+
+# Authenticate to an existing UC-enabled workspace.
+# Prefer `databricks auth login --host <workspace-url>` and omit explicit credentials,
+# or set host / token / Azure auth via environment variables.
+provider "databricks" {
+  host = var.databricks_host
+}

examples/adb-uc-external-location-file-events/terraform.tfvars.example

@@ -0,0 +1,8 @@
+databricks_host         = "https://adb-xxxxxxxxxxxx.azuredatabricks.net"
+name_prefix             = "file-events-demo"
+access_connector_id     = "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Databricks/accessConnectors/<name>"
+storage_account_id      = "/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Storage/storageAccounts/<name>"
+resource_group_name     = "<rg>"
+external_location_name  = "file-events-landing"
+external_location_url   = "abfss://landing@<storage>.dfs.core.windows.net/incoming"
+grant_principal         = "data-engineers"

examples/adb-uc-external-location-file-events/variables.tf

@@ -0,0 +1,42 @@
+variable "databricks_host" {
+  type        = string
+  description = "Workspace URL, e.g. https://adb-xxxx.azuredatabricks.net"
+}
+
+variable "name_prefix" {
+  type        = string
+  description = "Prefix for UC object names"
+  default     = "file-events-demo"
+}
+
+variable "access_connector_id" {
+  type        = string
+  description = "Azure resource ID of the Access Connector for Azure Databricks"
+}
+
+variable "storage_account_id" {
+  type        = string
+  description = "Azure resource ID of the ADLS Gen2 storage account"
+}
+
+variable "resource_group_name" {
+  type        = string
+  description = "Resource group containing the storage account"
+}
+
+variable "external_location_name" {
+  type        = string
+  description = "Name of the Unity Catalog external location"
+  default     = "file-events-landing"
+}
+
+variable "external_location_url" {
+  type        = string
+  description = "abfss:// URL for the external location path"
+}
+
+variable "grant_principal" {
+  type        = string
+  description = "UC group or user to grant on the credential and external location. Leave empty to skip grants."
+  default     = ""
+}

examples/adb-uc-external-location-file-events/versions.tf

@@ -0,0 +1,14 @@
+terraform {
+  required_version = ">= 1.3.0"
+
+  required_providers {
+    azurerm = {
+      source  = "hashicorp/azurerm"
+      version = ">= 3.0.0"
+    }
+    databricks = {
+      source  = "databricks/databricks"
+      version = ">= 1.50.0"
+    }
+  }
+}

modules/adb-uc-external-location-file-events/Makefile

@@ -0,0 +1,7 @@
+.PHONY: docs test_docs
+
+docs:
+	terraform-docs -c ../../.terraform-docs.yml .
+
+test_docs:
+	terraform-docs -c ../../.terraform-docs.yml --output-check .

modules/adb-uc-external-location-file-events/README.md

@@ -0,0 +1,133 @@
+# adb-uc-external-location-file-events
+
+Creates Azure Unity Catalog **storage credentials** and **external locations** with
+[managed file events](https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/manage-external-locations)
+enabled via Azure Queue Storage (managed AQS). Also assigns the documented Azure RBAC roles
+required for data access and automatic file-event setup.
+
+This is useful for [file arrival triggers](https://docs.databricks.com/aws/en/jobs/file-arrival-triggers)
+and other ingestion patterns that benefit from cloud storage change notifications.
+
+## Prerequisites
+
+- Unity Catalog–enabled Azure Databricks workspace
+- Existing Azure Data Lake Storage Gen2 account + container path
+- Existing [Access Connector for Azure Databricks](https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/azure-managed-identities)
+- Permission to assign Azure RBAC on the storage account and its resource group
+- Permission to create UC storage credentials and external locations
+
+## Azure RBAC (automatic managed file events)
+
+When `assign_azure_rbac = true` (default), the access connector managed identity receives:
+
+| Scope | Role | Purpose |
+| ----- | ---- | ------- |
+| Storage account | `Storage Blob Data Contributor` (or `Reader` if all locations are `read_only`) | Data plane access |
+| Storage account | `Storage Queue Data Contributor` | Subscribe to file-change notifications |
+| Storage account | `Storage Account Contributor` | Let Databricks auto-create the queue / routing |
+| Resource group | `EventGrid EventSubscription Contributor` | Let Databricks auto-create Event Grid subscriptions |
+
+`Storage Account Contributor` and the Event Grid role are only needed for **automatic** file-event
+setup. Manual / provided-queue configuration can omit them but is unsupported by Databricks.
+
+## Example
+
+```hcl
+module "uc_locations" {
+  source = "../../modules/adb-uc-external-location-file-events"
+
+  name_prefix          = "demo"
+  access_connector_id  = "/subscriptions/.../accessConnectors/uc-access-connector"
+  storage_account_id   = "/subscriptions/.../storageAccounts/stdemo"
+  resource_group_name  = "rg-demo"
+
+  external_locations = [
+    {
+      name    = "demo-landing"
+      url     = "abfss://landing@stdemo.dfs.core.windows.net/incoming"
+      comment = "Landing zone with file events"
+    }
+  ]
+
+  credential_grants = [
+    {
+      principal  = "data-engineers"
+      privileges = ["CREATE_EXTERNAL_LOCATION", "READ_FILES", "WRITE_FILES"]
+    }
+  ]
+
+  location_grants = [
+    {
+      principal  = "data-engineers"
+      privileges = ["BROWSE", "READ_FILES", "WRITE_FILES", "CREATE_EXTERNAL_TABLE", "CREATE_EXTERNAL_VOLUME"]
+    }
+  ]
+}
+```
+
+See also [examples/adb-uc-external-location-file-events](../../examples/adb-uc-external-location-file-events).
+
+<!-- BEGIN_TF_DOCS -->
+## Requirements
+
+| Name | Version |
+| ---- | ------- |
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.3.0 |
+| <a name="requirement_azurerm"></a> [azurerm](#requirement\_azurerm) | >= 3.0.0 |
+| <a name="requirement_databricks"></a> [databricks](#requirement\_databricks) | >= 1.50.0 |
+
+## Providers
+
+| Name | Version |
+| ---- | ------- |
+| <a name="provider_azurerm"></a> [azurerm](#provider\_azurerm) | >= 3.0.0 |
+| <a name="provider_databricks"></a> [databricks](#provider\_databricks) | >= 1.50.0 |
+
+## Modules
+
+No modules.
+
+## Resources
+
+| Name | Type |
+| ---- | ---- |
+| [azurerm_role_assignment.blob_data](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
+| [azurerm_role_assignment.eventgrid_subscription](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
+| [azurerm_role_assignment.queue_data](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
+| [azurerm_role_assignment.storage_account_contributor](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
+| [databricks_external_location.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/external_location) | resource |
+| [databricks_grants.credential](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/grants) | resource |
+| [databricks_grants.location](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/grants) | resource |
+| [databricks_storage_credential.this](https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/storage_credential) | resource |
+| [azurerm_client_config.current](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/client_config) | data source |
+| [azurerm_databricks_access_connector.this](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/databricks_access_connector) | data source |
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+| ---- | ----------- | ---- | ------- | :------: |
+| <a name="input_access_connector_id"></a> [access\_connector\_id](#input\_access\_connector\_id) | Azure resource ID of the Access Connector for Azure Databricks (managed identity used by the storage credential) | `string` | n/a | yes |
+| <a name="input_external_locations"></a> [external\_locations](#input\_external\_locations) | External locations to create. Each location gets managed AQS file events enabled. | <pre>list(object({<br/>    name      = string<br/>    url       = string<br/>    comment   = optional(string, "Managed by Terraform")<br/>    read_only = optional(bool, false)<br/>  }))</pre> | n/a | yes |
+| <a name="input_name_prefix"></a> [name\_prefix](#input\_name\_prefix) | Prefix used for storage credential and external location names | `string` | n/a | yes |
+| <a name="input_resource_group_name"></a> [resource\_group\_name](#input\_resource\_group\_name) | Resource group that contains the storage account (used for managed AQS and Event Grid RBAC) | `string` | n/a | yes |
+| <a name="input_storage_account_id"></a> [storage\_account\_id](#input\_storage\_account\_id) | Azure resource ID of the ADLS Gen2 storage account that backs the external location(s) | `string` | n/a | yes |
+| <a name="input_assign_azure_rbac"></a> [assign\_azure\_rbac](#input\_assign\_azure\_rbac) | Assign the documented Azure RBAC roles required for data access and automatic managed file events | `bool` | `true` | no |
+| <a name="input_create_storage_credential"></a> [create\_storage\_credential](#input\_create\_storage\_credential) | When true, create a storage credential backed by the access connector. When false, use existing\_credential\_name. | `bool` | `true` | no |
+| <a name="input_credential_grants"></a> [credential\_grants](#input\_credential\_grants) | UC grants on the storage credential. Defaults to empty (owner-only). | <pre>list(object({<br/>    principal  = string<br/>    privileges = list(string)<br/>  }))</pre> | `[]` | no |
+| <a name="input_existing_credential_name"></a> [existing\_credential\_name](#input\_existing\_credential\_name) | Name of an existing storage credential to reuse when create\_storage\_credential is false | `string` | `""` | no |
+| <a name="input_force_destroy"></a> [force\_destroy](#input\_force\_destroy) | Force destroy UC objects even if dependents exist | `bool` | `true` | no |
+| <a name="input_location_grants"></a> [location\_grants](#input\_location\_grants) | UC grants applied to every external location.<br/>Recommended privileges for data engineers: BROWSE, READ\_FILES, WRITE\_FILES,<br/>CREATE\_EXTERNAL\_TABLE, CREATE\_EXTERNAL\_VOLUME. | <pre>list(object({<br/>    principal  = string<br/>    privileges = list(string)<br/>  }))</pre> | `[]` | no |
+| <a name="input_storage_credential_name"></a> [storage\_credential\_name](#input\_storage\_credential\_name) | Name for the created storage credential. Defaults to "<name\_prefix>-storage-credential". | `string` | `""` | no |
+| <a name="input_subscription_id"></a> [subscription\_id](#input\_subscription\_id) | Azure subscription ID for managed AQS file-event configuration. Defaults to the current azurerm client subscription when empty. | `string` | `""` | no |
+
+## Outputs
+
+| Name | Description |
+| ---- | ----------- |
+| <a name="output_azure_rbac_roles"></a> [azure\_rbac\_roles](#output\_azure\_rbac\_roles) | Azure RBAC roles assigned to the access connector managed identity when assign\_azure\_rbac is true |
+| <a name="output_external_location_ids"></a> [external\_location\_ids](#output\_external\_location\_ids) | Map of external location name to ID |
+| <a name="output_external_location_names"></a> [external\_location\_names](#output\_external\_location\_names) | Names of the created external locations |
+| <a name="output_external_location_urls"></a> [external\_location\_urls](#output\_external\_location\_urls) | Map of external location name to URL |
+| <a name="output_storage_credential_id"></a> [storage\_credential\_id](#output\_storage\_credential\_id) | ID of the created storage credential (null when reusing an existing credential) |
+| <a name="output_storage_credential_name"></a> [storage\_credential\_name](#output\_storage\_credential\_name) | Name of the storage credential used by the external locations |
+<!-- END_TF_DOCS -->