From efc301463df61d2c436487e5e446818b017e1549 Mon Sep 17 00:00:00 2001 From: Kevin Hu Date: Sat, 4 Jul 2026 22:54:27 -0400 Subject: [PATCH 1/2] Document Job monitors (Data Job monitor type) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Job monitors (the "Job" monitor-creation type, API classification data-jobs) let customers alert on consecutive failed runs of an Airflow DAG, Databricks job, dbt job/model/test, Spark application, AWS Glue job, Azure Data Factory pipeline, or a custom-OpenLineage job. This capability has shipped in the product but had no public documentation. Content is grounded directly in the monitor-creation form (web-ui packages/apps/monitors/private/runtime/create/data-job/): the technology selector, tag filter, group-by behavior, consecutive-failure threshold (1-9999), auto-recovery on next success, and the default alert message template variables. Explicitly scoped to what the form actually supports today — there is no duration-based alert condition yet, only consecutive-failure count, so the docs do not claim one. Marked Preview per the DO Product Features tracker: the feature is gated behind a default-off per-org flag (data_jobs_monitoring_api_enabled) plus a frontend flag, and telemetry shows exactly 1 distinct org active in the last 30 days. Left a TODO for a real screenshot of the job-selection step (light and dark mode) — needs a human with product access. AI assistance: drafted by Claude Code, grounded in the monitor-creation form source and code-verified feature-flag/telemetry phase check; content follows the existing structure/style of this same doc page's Quality-monitor sections. --- .../en/monitors/types/data_observability.md | 51 ++++++++++++++++++- 1 file changed, 50 insertions(+), 1 deletion(-) diff --git a/content/en/monitors/types/data_observability.md b/content/en/monitors/types/data_observability.md index 4f13abba145..9204ccf89e1 100644 --- a/content/en/monitors/types/data_observability.md +++ b/content/en/monitors/types/data_observability.md @@ -1,6 +1,6 @@ --- title: Data Observability Monitor -description: "Monitor freshness, row count, column-level metrics, and custom SQL queries across your data warehouses." +description: "Monitor freshness, row count, column-level metrics, and custom SQL queries across your data warehouses, and monitor data job run failures." further_reading: - link: "/data_observability/" tag: "Documentation" @@ -8,6 +8,9 @@ further_reading: - link: "/data_observability/quality_monitoring/" tag: "Documentation" text: "Quality Monitoring" + - link: "/data_observability/jobs_monitoring/" + tag: "Documentation" + text: "Jobs Monitoring" - link: "/monitors/notify/" tag: "Documentation" text: "Configure your monitor notifications" @@ -271,6 +274,50 @@ Detect when a column's null percentage exceeds normal levels, which may indicate {{% /tab %}} {{< /tabs >}} +## Job monitors + +
Job monitors are in preview. Contact your Datadog representative or support to request access.
+ +Job monitors alert you when a data job reaches a threshold of consecutive failed runs. Job monitors require [Jobs Monitoring][11] to be set up with at least one supported job source. + +### Monitor creation + +To create a job monitor, navigate to {{< ui >}}Monitors{{< /ui >}} > {{< ui >}}New Monitor{{< /ui >}} > {{< ui >}}Job{{< /ui >}}. + +### Choose jobs to monitor + +1. Select a technology from the {{< ui >}}Select a technology{{< /ui >}} menu: {{< ui >}}Airflow{{< /ui >}}, {{< ui >}}AWS Glue{{< /ui >}}, {{< ui >}}Azure Data Factory{{< /ui >}}, {{< ui >}}Databricks{{< /ui >}}, {{< ui >}}dbt{{< /ui >}} (Job, Model, or Test), {{< ui >}}Spark{{< /ui >}}, or {{< ui >}}Custom Job{{< /ui >}} for jobs that emit [custom OpenLineage events][12]. + - If you select {{< ui >}}Custom Job{{< /ui >}}, also select the specific operation to monitor from the {{< ui >}}Select an operation{{< /ui >}} menu. +1. Optionally, narrow the jobs matched by this monitor using the {{< ui >}}Filter by tags{{< /ui >}} field (for example, `env:prod`). + +The preview panel shows the job names and recent run statistics that currently match your selection. + + + +### Configure alert conditions + +1. Under {{< ui >}}Define alerting context{{< /ui >}}, optionally add {{< ui >}}Group by{{< /ui >}} tags. Datadog automatically groups by `job_name`, plus additional tags specific to the selected technology (for example, `workspace_name` for Databricks, `project` for dbt). +1. Under {{< ui >}}Detection Method - Threshold{{< /ui >}}, set {{< ui >}}Trigger when{{< /ui >}} to the number of consecutive failed runs that should trigger an alert. + +The monitor automatically recovers after the next run succeeds. + +**Note**: Job monitors alert on consecutive run failures. They do not currently support alerting on job run duration. + +### Example notification + +{{< code-block lang="text" >}} +{{#is_alert}} +Databricks job **{{job_name.name}} ({{workspace_name.name}})** has reached {{value}} consecutive failures. + +Error: {{job.run_error}} + +[View in Databricks]({{job.run_url}}) +{{/is_alert}} +{{#is_recovery}} +Databricks job **{{job_name.name}}** recovered. +{{/is_recovery}} +{{< /code-block >}} + ## Annotate bounds For monitors using the **Anomaly** detection method, you can annotate bound ranges to provide feedback and improve the model over time. Unlike infrastructure metrics, data quality metrics are often business-specific, so use annotations to teach the model what behavior is normal for your data. @@ -300,3 +347,5 @@ On a monitor's status page, click {{< ui >}}Annotate Bounds{{< /ui >}}, select a [8]: /monitors/configuration/?tab=thresholdalert#thresholds [9]: /help/ [10]: /api/latest/data-observability/ +[11]: /data_observability/jobs_monitoring/ +[12]: /data_observability/jobs_monitoring/openlineage/ From 264687a5427066b8b66ca7d47dbb6b9b0c5d3ade Mon Sep 17 00:00:00 2001 From: Kevin Hu Date: Sat, 4 Jul 2026 23:13:12 -0400 Subject: [PATCH 2/2] Add Schema Change and Source-to-Target monitor documentation Extends this same PR (already documenting Job monitors) with the two remaining Data Observability monitor types found in DataObservabilityMonitorTypeSelector.tsx: Schema Change and Source-to-Target. Grounded directly in: - Backend: data-observability-schema-diff-worker (4 change types: added/removed/renamed/type_changed; hardcoded `> 0` threshold, no custom alert message support today) and data-observability-source-to-target-worker (absolute/percent diff, 30-minute match window between source and target metric computation). - Frontend: SchemaChangeDataSelection.tsx and DataQualitySourceToTargetEntitySelection.tsx for exact field labels. Both marked Preview: gated behind do-schema-diff/do-schema-change-monitors-ui and do-source-to-target-monitors respectively (default-off, per-org). Left two things explicitly unconfirmed rather than guessing: whether the UI exposes group-by columns for source-to-target monitors, and whether schema-change alerts support custom message templates (code suggests no, noted as such rather than a guess). AI assistance: drafted by Claude Code, grounded in the cited backend/ frontend source and the DO Product Features tracker's phase evidence. --- .../en/monitors/types/data_observability.md | 38 +++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/content/en/monitors/types/data_observability.md b/content/en/monitors/types/data_observability.md index 9204ccf89e1..1a98fe7a6b1 100644 --- a/content/en/monitors/types/data_observability.md +++ b/content/en/monitors/types/data_observability.md @@ -318,6 +318,44 @@ Databricks job **{{job_name.name}}** recovered. {{/is_recovery}} {{< /code-block >}} +## Schema change monitors + +
Schema change monitors are in preview. Contact your Datadog representative or support to request access.
+ +Schema change monitors alert you when a database, schema, table, or column is added, removed, renamed, or has its type changed. + +### Monitor creation + +To create a schema change monitor, navigate to {{< ui >}}Monitors{{< /ui >}} > {{< ui >}}New Monitor{{< /ui >}} > {{< ui >}}Schema Change{{< /ui >}}. + +### Choose data to monitor + +1. Select a warehouse from the {{< ui >}}Select warehouse{{< /ui >}} menu. +1. Use {{< ui >}}Choose entities within the warehouse to monitor{{< /ui >}} to filter by `account`, `database`, `schema`, `table`, or `column`. + +### Alert conditions + +Schema change monitors alert on any matching change (there is currently no configurable threshold). Each alert reports the affected entity's fully-qualified name, the change type (`added`, `removed`, `renamed`, or `type_changed`), and — for renames and type changes — the old and new values. + +**Note**: You cannot yet configure a custom alert message template for schema change monitors; alerts use a standard format grouped by change type and entity. + +## Source-to-target monitors + +
Source-to-target monitors are in preview. Contact your Datadog representative or support to request access.
+ +Source-to-target monitors compare a metric computed on a source table or column against the same metric computed on a target table or column — for example, to confirm that a row count in a source system matches the row count after it lands in a warehouse. Use them to catch data loss or drift introduced during a transformation or copy step. + +### Monitor creation + +1. Navigate to {{< ui >}}Monitors{{< /ui >}} > {{< ui >}}New Monitor{{< /ui >}} > {{< ui >}}Data Quality{{< /ui >}}, then select {{< ui >}}Source to Target{{< /ui >}}. +1. Choose a warehouse and a source table or column. +1. Choose a second warehouse and a target table or column. The target must be the same entity type as the source (table-to-table or column-to-column). +1. Select a metric type, following the same options as [Quality monitors](#select-your-metric-type). +1. Choose a comparison format: {{< ui >}}Difference{{< /ui >}} (`target - source`) or {{< ui >}}Percent Difference{{< /ui >}} (`(target - source) / source`). +1. Set an alert threshold using {{< ui >}}more than{{< /ui >}} or {{< ui >}}less than{{< /ui >}}. + +**Note**: Source and target metrics must be computed within 30 minutes of each other, or the comparison for that evaluation is skipped. + ## Annotate bounds For monitors using the **Anomaly** detection method, you can annotate bound ranges to provide feedback and improve the model over time. Unlike infrastructure metrics, data quality metrics are often business-specific, so use annotations to teach the model what behavior is normal for your data.