query_store plan/text fetch goes activity-driven: the store is the watermark (#2312)

CHANGELOG.md

@@ -87,6 +87,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **The force-plan recommendation now warns when the regressed query is parameter-sensitive** ([#2138] gap 3) - each regressed_queries row carries a `parameter_sensitivity_cofired` flag, computed inside the drill-down with the PARAMETER_SENSITIVITY detector's own thresholds (one cached plan whose per-execution cost varies >= 10x across parameter values, same floors, same window) joined by query hash - so the flag can never claim evidence the detector would not report. A flagged target's force-plan preview gains a caution block naming the risk (forcing pins ONE shape for every parameter value; the population that preferred the other plan inherits the wrong one permanently, quietly, because a forced plan no longer recompiles away) and the gentler first levers (statistics updates; PSP optimization / Query Store hints on 2022+), and the copy-paste surface gets a compact two-line version of the same warning. Unflagged targets render byte-identically to before. This flag is also the standing gate for the future auto-force bot: a flagged target is never auto-forced. Both SKUs, pinned by live tests in both stores.
 
 ### Fixed
+- **query_store's plan/text fetch is activity-driven: the store is the watermark** ([#2312]) - the collector's invariant 40-110s-per-run bill on big catalogs had a named mechanism at last: the #2210 watermark walk's daily expiry was supposed to be replaced by a re-verify cursor that was built, tested, documented, and **never wired** - so catalogs whose full walk needs more than a day expired MID-walk, restarted from plan_id 0, and looped the full catalog fetch forever (Finding 4). `TouchSql`, the liveness refresh the dimension GC depends on, had the same story: designed as 'the whole of the protection,' zero callers (Finding 3) - latent only because the perpetual walk's re-upserts accidentally stood in for it. The reshape retires all of it: the cycle's collected rows name their plans/texts, one touch-and-probe round trip refreshes map/dim liveness AND answers what the store lacks (plus per-cycle in-place-rewrite and Query-Store-reset detection via the live hashes the payload already carries), and the fetch selects exactly the missing ids under the same byte-budget arithmetic. A caught-up database issues NO target query - the measured 23s-to-discover-nothing becomes nothing; a Query Store reset recovers as the normal path instead of a special arm; a dormant plan resuming execution is fetched the cycle it resumes instead of waiting on a refresh horizon. V77 carries the schema strokes (nullable map digest for the content-less NULL-XML marker, `query_store_text.query_hash`, and wholesale deletion of the orphaned `planwm:`/`textwm:` state rows). Budget-deferred ids carry over in memory so a plan referenced once cannot be starved; ids the target no longer serves are dropped only on a provably-uncut pass. This also bends #2316's plan-dimension growth going forward: plans never referenced by a collected row are no longer shipped or stored at all.
 - **Darling Viewer crash on Queries -> Query Store by Duration** ([#2181], [#2331]) - the same uncatchable crash class as Lite's #2114, on the OTHER SKU: the grid's inline View Plan button referenced `DarkButton`, a key that IS defined in the Viewer - in `MainWindow.xaml`'s window resources, a scope a UserControl's templates cannot see, because StaticResource resolves lexically at load rather than through the runtime tree. The miss inside a cell template stack-overflows the process the moment the grid renders a row, which is also why it survived dogfooding: an EMPTY Query Store grid never applies its cell template. #2181 reported this against the Darling Viewer and was closed as a duplicate of the Lite fix on a wrong premise; #2331 re-proved it on 3.4.0. The button uses default chrome now (Lite's exact fix), and the XAML hygiene test's model is widened from per-app to per-FILE resolution (own keys + merged dictionaries + App.xaml scope - WPF's actual lookup), which flags exactly this class and produced zero false positives across both apps.
 - **The store self-metrics sweep's ~5-a-day "Exception while reading from stream" ERRORs were command timeouts in a network-fault costume** ([#2317]) - the sweep's sizing queries (`hypertable_detailed_size` across every hypertable - its inner `hypertable_local_size` is the frame the server log names - and `pg_database_size` over the whole store) ran on Npgsql's default 30-second timeout, which they outgrew under load on a 141-object store with a 100+ GB plan dimension. Npgsql enforces its deadline by cancelling the statement (the store side logs `canceling statement due to user request` - confirmed at the exact failure timestamps in the managed server's own log) and the client is left holding a torn stream, so the ERROR read as "the network broke" - the same misdirection #2294 named on the baseline path, one layer over. Every sweep statement now carries a five-minute timeout, and the worker caps the WHOLE sweep at the same five minutes through a linked cancellation (the sweep is awaited on the main loop, so five sequential statement timeouts must not stack into a 25-minute stall of per-server dispatch), the statement count is pinned to the timeout count so a sixth statement cannot ride the default back in, and the worker's catch names a timeout as a timeout - a sweep that still cannot finish skips the tick and the series gains a self-healing one-hour gap, deliberately NOT retrying into the same load.
 - **Gapped charts no longer bury their neighbours under opaque black fill** ([#2324]) - the #1944 gap markers (a NaN Y injected mid-gap so lines break across an outage) shipped first in 3.4.0, and they collide with the gradient area fill: reproduced headlessly against ScottPlot 5.1.59, one NaN in a FillY + ColorPositions series renders the ribbon as opaque black polygons with straight chord edges crossing the gap - the fill path closes its contours through the break, and its fill paint under ColorPositions is hardcoded black with the gradient shader expected to paint over it, which a NaN-bearing series defeats. On the reporter's dark theme that black buried every other series on every tab whose data had a collection gap; the one healthy tab was the one with gapless data. A gap-marked series now renders line-only - the break stays visible, nothing is buried - and continuous series keep the full gradient ribbon, pinned in both directions so the fix cannot quietly repeal the fill feature.

Darling/Darling.Tests/ActivityDrivenPlanFetchStoreTests.cs

@@ -0,0 +1,121 @@
+/*
+ * Copyright (c) 2026 Erik Darling, Darling Data LLC
+ *
+ * This file is part of the SQL Server Performance Monitor.
+ *
+ * Licensed under the MIT License. See LICENSE file in the project root for full license information.
+ */
+
+using System;
+using System.Linq;
+using PerformanceMonitor.Darling.Storage;
+using PerformanceMonitor.Darling.Viewer;
+using Xunit;
+
+namespace Darling.Tests;
+
+/// <summary>
+/// The V77 rung (#2312 Finding 2) — the schema strokes behind the activity-driven plan/text fetch: the
+/// plan map's <c>digest</c> goes nullable (the content-less marker for plans whose XML the engine cannot
+/// persist), <c>query_store_text</c> gains <c>query_hash</c> (the Query Store reset detector), and the
+/// retired <c>planwm:</c>/<c>textwm:</c> watermark state rows are deleted wholesale. These facts pin the
+/// rung's place on the ladder, the viewer probe's newest-first arm, and the migration SQL's load-bearing
+/// strokes. The fetch behavior itself is pinned in <c>QueryStorePlanFetchTests</c> and exercised live in
+/// the gated Postgres suites.
+/// </summary>
+public sealed class ActivityDrivenPlanFetchStoreTests
+{
+    /* ---------------- the rung ---------------- */
+
+    [Fact]
+    public void TheRungIsTheTopOfADenseLadder()
+    {
+        var versions = PgMigrations.Scripts.Select(s => s.Version).ToList();
+
+        Assert.Equal(77, versions.Max());
+        Assert.Equal(StorageVersion.SchemaVersion, versions.Max());
+        Assert.Equal(versions.Distinct().OrderBy(v => v), versions);
+
+        /* Dense above the one sanctioned historical hole at V45. */
+        var above = versions.Where(v => v > 45).OrderBy(v => v).ToList();
+        Assert.Equal(Enumerable.Range(above[0], above.Count), above);
+
+        Assert.Equal("activity-driven-plan-fetch", PgMigrations.Scripts.Single(s => s.Version == 77).Name);
+    }
+
+    /// <summary>The three strokes, each load-bearing and none allowed to drift out of the rung: without
+    /// the nullable digest the NULL-XML marker cannot land, without query_hash the reset detector has no
+    /// stored baseline, and without the deletes the orphaned watermark rows live forever (collector_state
+    /// has no retention, and the prune set no longer owns those prefixes).</summary>
+    [Fact]
+    public void TheRungCarriesAllThreeStrokes()
+    {
+        var sql = PgMigrations.Scripts.Single(s => s.Version == 77).Sql;
+
+        Assert.Contains("ALTER TABLE collect.query_store_plan_map ALTER COLUMN digest DROP NOT NULL", sql, StringComparison.Ordinal);
+        Assert.Contains("ALTER TABLE collect.query_store_text ADD COLUMN IF NOT EXISTS query_hash text", sql, StringComparison.Ordinal);
+        Assert.Contains("DELETE FROM collector_state WHERE collector_name = 'query_store_plan_xml' AND state_key LIKE 'planwm:%'", sql, StringComparison.Ordinal);
+        Assert.Contains("DELETE FROM collector_state WHERE collector_name = 'query_store_text' AND state_key LIKE 'textwm:%'", sql, StringComparison.Ordinal);
+    }
+
+    /* ---------------- the viewer probe ---------------- */
+
+    [Fact]
+    public void TheProbeMapsAFullyMigratedStoreTo77()
+    {
+        Assert.Equal(77, StorageVersion.SchemaVersion);
+        Assert.Equal(StorageVersion.SchemaVersion, ViewerDataService.RequiredStoreSchemaVersion);
+
+        /* 52 positional sentinels then the V77 one by name — the map takes 53 parameters. Present => 77,
+           newest-first; absent => the previous arm still answers 76 rather than falling through. */
+        var all = Enumerable.Repeat(true, 52).Cast<object>().ToArray();
+
+        Assert.Equal(77, InvokeMap(all, hasQueryStoreTextHash: true));
+        Assert.Equal(76, InvokeMap(all, hasQueryStoreTextHash: false));
+    }
+
+    [Fact]
+    public void TheProbeAsksForTheColumn_AndTheThreePlacesAgree()
+    {
+        Assert.Contains(
+            "table_name = 'query_store_text' AND column_name = 'query_hash'",
+            ViewerDataService.StoreSchemaProbeSql, StringComparison.Ordinal);
+
+        var mapParameters = typeof(ViewerDataService)
+            .GetMethod("MapProbedSchemaVersion", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Static)!
+            .GetParameters().Length;
+
+        var viewerSource = ReadViewerSource();
+
+        /* The reader must hand over exactly one argument per map parameter: ordinals are 0-based, so the
+           highest is Count - 1, and the next one up must NOT appear. */
+        Assert.Contains($"reader.GetBoolean({mapParameters - 1})", viewerSource, StringComparison.Ordinal);
+        Assert.DoesNotContain($"reader.GetBoolean({mapParameters})", viewerSource, StringComparison.Ordinal);
+    }
+
+    /* ---------------- helpers ---------------- */
+
+    private static int InvokeMap(object[] leading, bool hasQueryStoreTextHash)
+    {
+        var method = typeof(ViewerDataService)
+            .GetMethod("MapProbedSchemaVersion", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Static)!;
+
+        var args = leading.Concat(new object[] { hasQueryStoreTextHash }).ToArray();
+        Assert.Equal(method.GetParameters().Length, args.Length);
+
+        return (int)method.Invoke(null, args)!;
+    }
+
+    private static string ReadViewerSource([System.Runtime.CompilerServices.CallerFilePath] string thisFile = "")
+    {
+        var dir = System.IO.Path.GetDirectoryName(thisFile)!;
+        var relative = System.IO.Path.Combine("Darling", "PerformanceMonitor.Darling.Viewer", "ViewerDataService.cs");
+        while (dir is not null && !System.IO.File.Exists(System.IO.Path.Combine(dir, relative)))
+        {
+            dir = System.IO.Path.GetDirectoryName(dir);
+        }
+
+        Assert.NotNull(dir);
+        return System.IO.File.ReadAllText(System.IO.Path.Combine(dir!, relative));
+    }
+}

Darling/Darling.Tests/AzureForeignStatePruneTests.cs

@@ -125,17 +125,15 @@ public void ADatabaseNamedRegistrationIsPruned(string catalog)
     [Fact]
     public void BothArmsPruneEveryPerDatabasePrefix()
     {
-        Assert.Equal(5, QueryStorePerDatabaseState.PrunableKeys.Count);
-        Assert.Contains(QueryStorePerDatabaseState.PrunableKeys,
-            k => k.Prefix == QueryStorePlanXmlState.WatermarkKeyPrefix);
+        /* #2312 shrank this from five to three: the planwm:/textwm: watermark families retired with the
+           watermarks themselves (the fetches are activity-driven against the store now), and V77 deleted
+           their orphaned rows wholesale — a dropped-database prune has nothing left to own there. */
+        Assert.Equal(3, QueryStorePerDatabaseState.PrunableKeys.Count);
         Assert.Contains(QueryStorePerDatabaseState.PrunableKeys,
             k => k.Prefix == QueryStoreBackfillState.DoneKeyPrefix);
         Assert.Contains(QueryStorePerDatabaseState.PrunableKeys,
             k => k.Prefix == QueryStoreBackfillState.HoleKeyPrefix);
-        /* #2150: the text watermark, keyed prefix + databaseName exactly like the plan watermark. */
-        Assert.Contains(QueryStorePerDatabaseState.PrunableKeys,
-            k => k.Prefix == QueryStoreTextState.WatermarkKeyPrefix);
-        /* #2312: the open-interval refresh stamp, the fifth per-database prefix. */
+        /* #2312: the open-interval refresh stamp. */
         Assert.Contains(QueryStorePerDatabaseState.PrunableKeys,
             k => k.Prefix == QueryStoreOpenIntervalState.WatermarkKeyPrefix);
 

Darling/Darling.Tests/DarlingDimensionGcBoundTests.cs

@@ -144,32 +144,9 @@ public void NeitherPruneOrder_CanLeaveAMapRowResolvingToAnAbsentDigest(int factR
         }
     }
 
-    /// <summary>
-    /// #2210: the re-verify cursor paces itself off <c>RefreshAfter</c> and NEVER touches the watermark. The
-    /// slice is a row count over an id range, which is the whole point — the old expiry walked BYTES and could
-    /// not finish inside a day on the catalogs that mattered (15.9 to 107.5 hours measured), so those restarted
-    /// forever. Redstone's 77k ids at a 5-minute cadence over a 1-day sweep is ~267 ids per pass.
-    /// </summary>
-    [Fact]
-    public void CursorSlice_PacesASweepWithinTheRefreshPeriod_AndNeverReturnsZeroForALiveCatalog()
-    {
-        var day = TimeSpan.FromDays(1);
-        var cadence = TimeSpan.FromMinutes(5);
-
-        var redstone = QueryStorePlanMap.CursorSliceWidth(77_176, day, cadence);
-        Assert.InRange(redstone, 200, 350);
-
-        /* A sweep must actually cover the range within the period: slice * passes >= watermark. */
-        var passes = day.Ticks / cadence.Ticks;
-        Assert.True(redstone * passes >= 77_176, "the sweep must cover the id range inside one refresh period");
-
-        /* Never zero for a live catalog, and never wider than the range itself. */
-        Assert.True(QueryStorePlanMap.CursorSliceWidth(10, day, cadence) > 0);
-        Assert.Equal(10, QueryStorePlanMap.CursorSliceWidth(10, day, cadence));
-
-        /* A fresh database has no watermark to re-verify, so there is nothing to slice. */
-        Assert.Equal(0, QueryStorePlanMap.CursorSliceWidth(0, day, cadence));
-    }
+    /* #2312: the CursorSlice facts that sat here are gone with the cursor itself — it was designed in
+       #2210 and never wired (Finding 4), and the in-place-rewrite job it existed for now runs per-cycle
+       through TouchAndProbeSql's hash comparison, pinned in QueryStorePlanFetchTests. */
 
     /// <summary>
     /// #2210: the DIMENSION must outlive the MAP, expressed the way it actually matters — as cutoff DATES from

Darling/Darling.Tests/PlanContentRetentionTests.cs

@@ -264,9 +264,10 @@ private static int InvokeMap(object[] leading, bool hasPlanContentRetentionKnob)
         var method = typeof(ViewerDataService)
             .GetMethod("MapProbedSchemaVersion", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Static)!;
 
-        /* #2319 appended hasQueryStoreHealth after this rung's parameter — pass it FALSE so these
-           facts keep exercising the V75/V74 arms rather than the newer one. */
-        var args = leading.Concat(new object[] { hasPlanContentRetentionKnob, false }).ToArray();
+        /* #2319 appended hasQueryStoreHealth and #2312 appended hasQueryStoreTextHash after this rung's
+           parameter — pass both FALSE so these facts keep exercising the V75/V74 arms rather than the
+           newer ones. */
+        var args = leading.Concat(new object[] { hasPlanContentRetentionKnob, false, false }).ToArray();
         Assert.Equal(method.GetParameters().Length, args.Length);
 
         return (int)method.Invoke(null, args)!;