[WIP] Document Job, Schema Change, and Source-to-Target monitor types#37959
Draft
kevinzenghu wants to merge 2 commits into
Draft
[WIP] Document Job, Schema Change, and Source-to-Target monitor types#37959kevinzenghu wants to merge 2 commits into
kevinzenghu wants to merge 2 commits into
Conversation
Job monitors (the "Job" monitor-creation type, API classification data-jobs) let customers alert on consecutive failed runs of an Airflow DAG, Databricks job, dbt job/model/test, Spark application, AWS Glue job, Azure Data Factory pipeline, or a custom-OpenLineage job. This capability has shipped in the product but had no public documentation. Content is grounded directly in the monitor-creation form (web-ui packages/apps/monitors/private/runtime/create/data-job/): the technology selector, tag filter, group-by behavior, consecutive-failure threshold (1-9999), auto-recovery on next success, and the default alert message template variables. Explicitly scoped to what the form actually supports today — there is no duration-based alert condition yet, only consecutive-failure count, so the docs do not claim one. Marked Preview per the DO Product Features tracker: the feature is gated behind a default-off per-org flag (data_jobs_monitoring_api_enabled) plus a frontend flag, and telemetry shows exactly 1 distinct org active in the last 30 days. Left a TODO for a real screenshot of the job-selection step (light and dark mode) — needs a human with product access. AI assistance: drafted by Claude Code, grounded in the monitor-creation form source and code-verified feature-flag/telemetry phase check; content follows the existing structure/style of this same doc page's Quality-monitor sections.
Contributor
Preview links (active after the
|
Extends this same PR (already documenting Job monitors) with the two remaining Data Observability monitor types found in DataObservabilityMonitorTypeSelector.tsx: Schema Change and Source-to-Target. Grounded directly in: - Backend: data-observability-schema-diff-worker (4 change types: added/removed/renamed/type_changed; hardcoded `> 0` threshold, no custom alert message support today) and data-observability-source-to-target-worker (absolute/percent diff, 30-minute match window between source and target metric computation). - Frontend: SchemaChangeDataSelection.tsx and DataQualitySourceToTargetEntitySelection.tsx for exact field labels. Both marked Preview: gated behind do-schema-diff/do-schema-change-monitors-ui and do-source-to-target-monitors respectively (default-off, per-org). Left two things explicitly unconfirmed rather than guessing: whether the UI exposes group-by columns for source-to-target monitors, and whether schema-change alerts support custom message templates (code suggests no, noted as such rather than a guess). AI assistance: drafted by Claude Code, grounded in the cited backend/ frontend source and the DO Product Features tracker's phase evidence.
kevinzenghu
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documents the three Data Observability monitor types found in
DataObservabilityMonitorTypeSelector.tsxthat had zero public documentation:data-jobs, GA) — alert on consecutive failed runs for Airflow, Glue, ADF, Databricks, dbt, Spark, or custom-OpenLineage jobs.Evidence
web-ui/packages/apps/monitors/.../create/data-job/— see original commit for full citation list.data-observability-schema-diff-worker(backend, 4 change types, hardcoded>0threshold) +SchemaChangeDataSelection.tsx(frontend field labels).data-observability-source-to-target-worker(absolute/percent diff, 30-min match window) +DataQualitySourceToTargetEntitySelection.tsx.GA— see DO Product Features tracker (this PR documents it as Preview, consistent with the tracker's corrected phase and the underlyingdata_jobs_monitoring_api_enableddefault-off flag). Schema Change and Source-to-Target are Preview per their own default-off per-org flags (do-schema-diff+do-schema-change-monitors-ui,do-source-to-target-monitors).Open items
Status
Work in progress — not ready for review yet. Draft, no reviewers requested.
AI assistance
Content and phase verification drafted by Claude Code, grounded in the frontend/backend source cited above — flagging per the AI-assistance disclosure in CONTRIBUTING.md.