From a7502272cf79909cf70539d60a2fff58a3d1a803 Mon Sep 17 00:00:00 2001 From: Kevin Hu Date: Sat, 4 Jul 2026 23:41:16 -0400 Subject: [PATCH 1/2] Add Azure Data Factory Jobs Monitoring documentation (preview) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Corrects an earlier finding: a prior code-only pass found an internal knowledge-base comment stating "not GA, do not promise to customers" and concluded this shouldn't be publicly documented. Kevin confirmed directly that ADF is in private preview with access available on request, backed by a setup guide — the internal knowledge-base file was stale relative to the feature's actual current phase (a real instance of the "most recent live signal wins" precedence rule; the stale knowledge-base file should be updated separately). Content is adapted from that setup guide into house style (tabs for the two role-assignment options, standard Datadog UI/CLI conventions), following the existing AWS Glue Jobs Monitoring page's structure. No internal-only URLs or infrastructure in this one — Azure Portal, az CLI, and Datadog's own already-public UI paths only. AI assistance: adapted by Claude Code from a human-authored internal setup doc (linked in the PR), restructured into Datadog documentation style; Azure permission/role details are not independently verified against code, only against the source doc. --- .../jobs_monitoring/_index.md | 1 + .../jobs_monitoring/azure_data_factory.md | 145 ++++++++++++++++++ 2 files changed, 146 insertions(+) create mode 100644 content/en/data_observability/jobs_monitoring/azure_data_factory.md diff --git a/content/en/data_observability/jobs_monitoring/_index.md b/content/en/data_observability/jobs_monitoring/_index.md index 08c0454054c..2548f615108 100644 --- a/content/en/data_observability/jobs_monitoring/_index.md +++ b/content/en/data_observability/jobs_monitoring/_index.md @@ -38,6 +38,7 @@ Data Observability: Jobs Monitoring supports multiple job technologies. To get s {{< image-card href="/data_jobs/airflow/" src="integrations_logos/airflow.png" alt="Airflow" >}} {{< image-card href="/data_observability/jobs_monitoring/dbt" src="integrations_logos/dbt-cloud_large.svg" alt="dbt cloud" >}} {{< image-card href="/data_observability/jobs_monitoring/glue/" src="integrations_logos/amazon_glue.png" alt="AWS Glue" >}} + {{< image-card href="/data_observability/jobs_monitoring/azure_data_factory/" src="integrations_logos/azure_data_factory.png" alt="Azure Data Factory" >}} {{< /card-grid >}}
diff --git a/content/en/data_observability/jobs_monitoring/azure_data_factory.md b/content/en/data_observability/jobs_monitoring/azure_data_factory.md new file mode 100644 index 00000000000..454d0d733af --- /dev/null +++ b/content/en/data_observability/jobs_monitoring/azure_data_factory.md @@ -0,0 +1,145 @@ +--- +title: "Jobs Monitoring for Azure Data Factory" +description: "Enable Data Observability: Jobs Monitoring for Azure Data Factory pipelines with Datadog." +further_reading: + - link: '/data_jobs' + tag: 'Documentation' + text: 'Data Observability: Jobs Monitoring' + - link: '/getting_started/integrations/azure/' + tag: 'Documentation' + text: 'Getting Started with Azure' +--- + +
Jobs Monitoring for Azure Data Factory is in preview. Contact your Datadog representative or support to request access.
+ +## Overview + +The Azure Data Factory (ADF) integration provides visibility into your ADF pipeline runs, activities, and data lineage across your Azure environment. + +## Prerequisites + +- Complete the [Datadog Azure integration setup][1]. The ADF integration reuses the App Registration (service principal) that the Azure integration already manages. +- The App Registration must have monitoring access to the subscriptions containing your Data Factories. + +**Note**: Azure Data Factory Jobs Monitoring does not work with the [Azure Native Integration][2]. + +## Setup + +### Step 1: Grant additional Azure permissions + +The Datadog Azure integration assigns the built-in **Monitoring Reader** role to your App Registration at subscription scope. This covers read-only Azure Resource Manager (ARM) endpoints, but not the three POST endpoints Datadog needs to enumerate pipeline run history: + +| Endpoint | Required permission | +|---|---| +| `POST .../queryPipelineRuns` | `Microsoft.DataFactory/factories/querypipelineruns/action` | +| `POST .../pipelineruns/{id}/queryActivityRuns` | `Microsoft.DataFactory/factories/pipelineruns/queryactivityruns/action` | +| `POST .../queryTriggerRuns` | `Microsoft.DataFactory/factories/querytriggerruns/action` | + +Without these permissions, Datadog can inventory your factories but can't collect pipeline run history. Add one of the following role assignments to the Datadog App Registration for each Data Factory you want to monitor. + +By default, Datadog monitors all Data Factories within the subscriptions you select in [Step 2](#step-2-enable-azure-data-factory-in-datadog). To restrict monitoring to specific factories, scope the role assignment to each factory instead of the subscription. + +{{< tabs >}} +{{% tab "Minimal custom role (recommended)" %}} + +This grants only the permissions Datadog needs, with no write access to your factories. + +1. Save the following JSON to a file named `datadog-adf-reader.json`, replacing `` with your Azure subscription ID: + + ```json + { + "Name": "Datadog Data Observability - ADF Reader", + "Description": "Read-only access to ADF factory metadata and run history for Datadog Data Observability.", + "IsCustom": true, + "Actions": [ + "Microsoft.DataFactory/factories/read", + "Microsoft.DataFactory/factories/*/read", + "Microsoft.DataFactory/factories/querypipelineruns/action", + "Microsoft.DataFactory/factories/querytriggerruns/action", + "Microsoft.DataFactory/factories/pipelineruns/queryactivityruns/action" + ], + "NotActions": [], + "DataActions": [], + "NotDataActions": [], + "AssignableScopes": ["/subscriptions/"] + } + ``` + +1. Open [Azure Cloud Shell][3] and run the following, replacing the placeholder values: + + ```shell + az role definition create --role-definition ./datadog-adf-reader.json + + az role assignment create \ + --assignee \ + --role "Datadog Data Observability - ADF Reader" \ + --scope /subscriptions//resourceGroups//providers/Microsoft.DataFactory/factories/ + ``` + + Repeat the `az role assignment create` command for each Data Factory you want to monitor. + +**Note**: To find the Datadog App Registration Client ID, go to [Integrations > Azure][4] in Datadog and locate the App Registration entry, or find it in the Azure Portal under **Microsoft Entra ID > App registrations > [your Datadog app] > Overview**. + +**Note**: Creating a custom role requires the **Owner** or **User Access Administrator** role at the assignable scope. If your security policy prohibits custom roles, use the built-in role instead. + +{{% /tab %}} +{{% tab "Built-in Data Factory Contributor role" %}} + +This is simpler to configure, but grants write access to the factory in addition to the read permissions Datadog needs. Scope the assignment to the specific factory (rather than the subscription or resource group) to limit that risk. + +In the Azure Portal: + +1. Navigate to your Data Factory. +1. Select {{< ui >}}Access Control (IAM){{< /ui >}} from the left sidebar. +1. Click {{< ui >}}Add{{< /ui >}} > {{< ui >}}Add role assignment{{< /ui >}}. +1. Select the **Data Factory Contributor** role. +1. Under {{< ui >}}Members{{< /ui >}}, search for and select your Datadog App Registration. +1. Click {{< ui >}}Review + assign{{< /ui >}}. + +Or using the Azure CLI: + +```shell +az role assignment create \ + --assignee \ + --role "Data Factory Contributor" \ + --scope /subscriptions//resourceGroups//providers/Microsoft.DataFactory/factories/ +``` + +{{% /tab %}} +{{< /tabs >}} + +### Step 2: Enable Azure Data Factory in Datadog + +1. In Datadog, navigate to [Data Observability > Settings > Integrations][5]. +1. Locate the {{< ui >}}Azure Data Factory{{< /ui >}} section and click {{< ui >}}Configure{{< /ui >}}. +1. On the onboarding page, select the Azure App Registration you granted permissions to in Step 1. If you haven't connected an Azure App Registration to Datadog yet, complete the [Azure integration setup][1] first. +1. Select the subscriptions that contain the Data Factories you want to monitor. Only subscriptions accessible to the selected App Registration are shown. + + **Note**: You must select at least one subscription to save the configuration. Datadog monitors all Data Factories within the selected subscriptions. + +1. Click {{< ui >}}Save{{< /ui >}}. + +### Step 3: View your pipeline runs + +After saving, Datadog schedules an inventory crawl of your selected subscriptions. Once the crawl completes, your ADF pipeline runs appear on the [Jobs page][6], listed with status, duration, and associated factory and subscription. Drill into an individual pipeline run to see activity-level details and data lineage for Copy activities. + +If you don't see your pipeline runs after a few hours, check the following: + +- The App Registration has the role assignment from Step 1 on each target factory: + + ```shell + az role assignment list \ + --assignee \ + --scope /subscriptions//resourceGroups//providers/Microsoft.DataFactory/factories/ + ``` + +- The subscriptions containing your factories are selected in the integration configuration (Step 2). +- A Datadog representative has enabled preview access for your organization. +- Your factories have had pipeline runs recently. The first crawl only covers the hour before it ran; later crawls pick up where the previous one left off. Runs completed before you configured the integration don't appear retroactively. + +[1]: /getting_started/integrations/azure/ +[2]: /integrations/guide/azure-native-integration +[3]: https://shell.azure.com +[4]: https://app.datadoghq.com/integrations/azure +[5]: https://app.datadoghq.com/data-obs/settings/integrations +[6]: https://app.datadoghq.com/data-obs/jobs From b4f6606a64993373881e3fc441e2791d5b2dee2f Mon Sep 17 00:00:00 2001 From: Kevin Hu Date: Sun, 5 Jul 2026 01:08:03 -0400 Subject: [PATCH 2/2] Add Azure Data Factory to the Jobs Monitoring sidebar navigation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Same class of bug just found and fixed for Airbyte: this PR's new page and card-grid link were correctly added, but the separate sidebar-nav config (config/_default/menus/main.en.yaml) was not — this branch predates discovering that file exists. Added the missing entry right after AWS Glue, matching the card-grid ordering already in this PR. --- config/_default/menus/main.en.yaml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/config/_default/menus/main.en.yaml b/config/_default/menus/main.en.yaml index 69ba6d57d55..3a8e8bed194 100644 --- a/config/_default/menus/main.en.yaml +++ b/config/_default/menus/main.en.yaml @@ -5442,6 +5442,11 @@ menu: identifier: jobs_monitoring_glue parent: data_jobs weight: 650000 + - name: Azure Data Factory + url: data_observability/jobs_monitoring/azure_data_factory + identifier: jobs_monitoring_azure_data_factory + parent: data_jobs + weight: 660000 - name: Custom Jobs (OpenLineage) url: data_observability/jobs_monitoring/openlineage identifier: openlineage_integrations