From 85e91eb3f3835a9ef235ef46529f7cc00d85b201 Mon Sep 17 00:00:00 2001 From: Oleg <18466855+EvaTheSalmon@users.noreply.github.com> Date: Tue, 20 Oct 2026 11:57:02 +0300 Subject: [PATCH] feat(data): raise the archive state from the catalogue The status bar's archive indicator and its ConnectionLost entry came from the live-edge poll, which a later slice replaces with OPC UA. The poll also turned any failure, a missing trends table included, into ConnectionLost, and the status bar and the read loop each wrote their own entry for one outage. PenCatalogueSync now publishes ArchiveConnectionState through a sink of ArchiveReadLoop; MarkerSync passes none. The loop raises ConnectionLost after three connection-class failures in a row and Connected on the first success and the first success after a fault. Connection class is ArchiveFault.Unreachable, widened to SQLSTATE class 08, 57P01-57P03 and 53300 so a restart's recovery is not ReadFailed. Other kinds keep their own entries; one outage makes one entry from the loops and the bar. The catalogue command gets a 5 s bound instead of the 300 s backstop, and the window reads the catalogue at open, so the first Connected and the failure run start at window open. IDataProvider.ConnectionFaults, its republish in TrendCoordinator and the poll's state machine are deleted; the poll keeps its statements and samples. --- AGENTS.md | 20 +- .../Data/ArchiveConnectionState.cs | 5 +- .../SemiPlot.Core/Data/Errors/ArchiveError.cs | 2 +- .../SemiPlot.Core/Data/Errors/ArchiveFault.cs | 9 +- SemiPlot/SemiPlot.Core/Data/IDataProvider.cs | 5 - .../ArchiveExceptionMapper.cs | 15 + .../PostgresConnectionSettings.cs | 2 +- .../PostgresDataProvider.cs | 59 +- .../RealtimePoll.cs | 71 +- .../ArchiveProviderFactory.cs | 22 +- .../SemiPlot.Tests.Integration/ArmedGate.cs | 123 +++- .../CatalogueReadUnreachableTests.cs | 34 + .../Journeys/LiveEdgeArchiveJourneyTests.cs | 10 +- .../RealtimeArchive.cs | 21 - .../RealtimeEmptyArchiveTests.cs | 35 +- .../RealtimePollReadTests.cs | 67 +- .../RealtimePollTests.cs | 41 +- .../RealtimeSubscriptionDisposalTests.cs | 21 +- .../RealtimeSubscriptionTests.cs | 22 +- .../UnreachableArchive.cs | 7 +- .../Postgres/ArchiveExceptionMapperTests.cs | 27 + .../Postgres/ArchiveStatementTextTests.cs | 17 +- .../Postgres/PostgresCompositionTests.cs | 27 - .../UI/Bridge/FakeDataProvider.cs | 31 +- .../UI/Bridge/PenCatalogueSyncTests.cs | 220 +++++- .../UI/Bridge/TrendCoordinatorTests.cs | 93 +-- .../UI/Chart/TrendChartViewModelTests.cs | 15 - .../UI/Di/TrendWindowBuildTests.cs | 23 +- .../MainWindow/AppStatusBarViewModelTests.cs | 6 +- .../UI/MainWindow/MainWindowTestBuilder.cs | 2 +- .../UI/MainWindow/MainWindowViewModelTests.cs | 9 +- .../UI/MainWindow/TrendWindowTests.cs | 70 +- .../UI/Markers/MarkerSyncTests.cs | 24 +- .../UI/Startup/AppMainWindowTests.cs | 2 - .../SemiPlot.UI/Bridge/ArchiveReadLoop.cs | 60 +- .../SemiPlot.UI/Bridge/PenCatalogueSync.cs | 19 +- .../SemiPlot.UI/Bridge/TrendCoordinator.cs | 20 +- .../SemiPlot.UI/Localization/Resources.resx | 4 +- .../Localization/Resources.ru.resx | 4 +- .../MainWindow/AppStatusBarViewModel.cs | 8 +- .../SemiPlot.UI/MainWindow/TrendWindow.cs | 24 +- SemiPlot/SemiPlot.UI/Markers/MarkerSync.cs | 7 +- .../Messages/ArchiveFailureMapper.cs | 4 +- docs/architecture/charting.md | 15 +- docs/architecture/data-integration.md | 140 ++-- docs/architecture/overview.md | 74 +- docs/architecture/testing-strategy.md | 12 +- docs/architecture/trend-interaction.md | 11 +- .../20261019-archive-state-from-catalogue.md | 671 ++++++++++++++++++ 49 files changed, 1585 insertions(+), 645 deletions(-) create mode 100644 SemiPlot/SemiPlot.Tests.Integration/CatalogueReadUnreachableTests.cs create mode 100644 docs/plans/completed/20261019-archive-state-from-catalogue.md diff --git a/AGENTS.md b/AGENTS.md index 013a406..cd03c11 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -188,8 +188,8 @@ Frame cost during a drag is measured, not asserted: `docs/architecture/testing-s ### Container tests Every test of the `ArchiveDatabaseCollection` in `SemiPlot.Tests.Integration` needs a container runtime and -nothing else. The socket classes (`RealtimePollTests`, `RealtimeSubscriptionDisposalTests`, -`ConvergeReadinessTests`) carry no collection, take no fixture and run without a runtime. +nothing else. The socket classes (`RealtimePollTests`, `CatalogueReadUnreachableTests`, +`RealtimeSubscriptionDisposalTests`, `ConvergeReadinessTests`) carry no collection, take no fixture and run without a runtime. `SemiPlot/bench/Dockerfile` copies `/semibase` out of `ghcr.io/semiteq/semibase:latest`, which the fixture pulls ahead of the build, and runs `semibase bench` from `/docker-entrypoint-initdb.d/` before the published port opens. The container @@ -358,10 +358,12 @@ No abbreviations in names. `Pens and groups`, `docs/architecture/overview.md#the-pen-and-group-editor`), so a control added to the row fails the suite rather than quietly shipping a second editor. - A stored pen change reaches the running chart through `Bridge/PenCatalogueSync` and no other route: it - re-reads `IDataProvider.QueryPensAsync` every 5 s and on `ReadNow()` after each write the editor lands, - and `MainWindow/PenCatalogueApplier` hands each read to the chart, which compares it with the pens it - shows, then rebuilds the sidebar. The chart is never rebuilt whole - (`docs/architecture/overview.md#the-live-catalogue`). + re-reads `IDataProvider.QueryPensAsync` once when the window opens, every 5 s and on `ReadNow()` after + each write the editor lands, and `MainWindow/PenCatalogueApplier` hands each read to the chart, which + compares it with the pens it shows, then rebuilds the sidebar. The chart is never rebuilt whole + (`docs/architecture/overview.md#the-live-catalogue`). The same reads are the source of the archive + connection state the status bar shows, `PenCatalogueSync.ConnectionStates` + (`docs/architecture/data-integration.md#the-connection-state-the-catalogue-read-reports`). - A stored time marker reaches the running chart through `Markers/MarkerSync` and no other route: it re-reads `IMarkerStore.ReadAsync` every 5 s and on `ReadNow()` after each marker write, and `Chart/ChartMarkers` reconciles each read into `Core/Trends/MarkerSet`, which keeps a marker @@ -379,7 +381,11 @@ No abbreviations in names. at the call site. The one message with no error behind it is `AppStatusBarViewModel.ConnectionRestored`, which builds its own `Info` view because the mapper maps errors. A `catch`, or an Rx `onError`, that only logs is a defect - (`docs/architecture/data-integration.md#no-failure-stops-at-the-log`). Code-behind reaches the + (`docs/architecture/data-integration.md#no-failure-stops-at-the-log`). A failure for lack of a + connection (`ArchiveFault.Unreachable`) in the catalogue loop, the marker loop or the live-edge poll is + logged by that reader and reaches the panel through the status bar's one `ConnectionLost` entry, which + the catalogue reads raise; the three cases that reach only the log are listed in that same section. + Code-behind reaches the panel through its view model (`TrendChartViewModel.ReportFailure`, `MainWindowViewModel.ReportFailure`, `PenEditorViewModel.ReportFailure`, `StartupFailureViewModel.ReportFailure`, `AxisScalePanelViewModel.ReportFailure`, `TimeWindowPanelViewModel.ReportFailure`, diff --git a/SemiPlot/SemiPlot.Core/Data/ArchiveConnectionState.cs b/SemiPlot/SemiPlot.Core/Data/ArchiveConnectionState.cs index 7f18a8e..0c88846 100644 --- a/SemiPlot/SemiPlot.Core/Data/ArchiveConnectionState.cs +++ b/SemiPlot/SemiPlot.Core/Data/ArchiveConnectionState.cs @@ -3,9 +3,8 @@ namespace SemiPlot.Core.Data; /// -/// What a provider reports about its own connection to the archive. is null while the -/// archive answers and names the fault while it does not; never filter with DistinctUntilChanged, -/// every subscription's first tick reports . +/// What a reader of the archive reports about its connection to it. is null while the +/// archive answers and names the fault while it does not. /// public sealed record ArchiveConnectionState(ArchiveError? Fault) { diff --git a/SemiPlot/SemiPlot.Core/Data/Errors/ArchiveError.cs b/SemiPlot/SemiPlot.Core/Data/Errors/ArchiveError.cs index e7ccbaf..30205c7 100644 --- a/SemiPlot/SemiPlot.Core/Data/Errors/ArchiveError.cs +++ b/SemiPlot/SemiPlot.Core/Data/Errors/ArchiveError.cs @@ -35,7 +35,7 @@ private static string Describe(ArchiveFault kind, string host, int port, string $"The {archive} does not carry the columns SemiPlot reads (SQLSTATE 42703): {detail}", ArchiveFault.QueryTimedOut => $"The server ended a statement against the {archive} (SQLSTATE 57014).", ArchiveFault.ConnectionLost => - $"The live edge of the {archive} stopped answering after {detail} consecutive failed reads.", + $"The {archive} stopped answering after {detail} consecutive failed reads.", ArchiveFault.ValueRejected => $"The {archive} refused a value written to pen '{detail}' (SQLSTATE 23514).", ArchiveFault.NameTaken => $"The {archive} already holds a group named '{detail}' (SQLSTATE 23505).", diff --git a/SemiPlot/SemiPlot.Core/Data/Errors/ArchiveFault.cs b/SemiPlot/SemiPlot.Core/Data/Errors/ArchiveFault.cs index 55844da..69f68b0 100644 --- a/SemiPlot/SemiPlot.Core/Data/Errors/ArchiveFault.cs +++ b/SemiPlot/SemiPlot.Core/Data/Errors/ArchiveFault.cs @@ -6,7 +6,10 @@ namespace SemiPlot.Core.Data.Errors; /// public enum ArchiveFault { - /// No connection: a refused or reset socket, a client bound firing, a host that does not answer. + /// + /// No connection: a refused or reset socket, a client bound firing, a host that does not answer, or a server + /// that refuses or ends the session (SQLSTATE class 08, 57P01-57P03, 53300). + /// Unreachable, /// The server answered and refused the credentials or a grant (28P01, 28000, 42501). Detail is the username. @@ -24,7 +27,9 @@ public enum ArchiveFault /// The server ended a statement (57014): a statement_timeout or an administrator's cancel. QueryTimedOut, - /// A run of consecutive poll ticks failed. Detail is the number of failures that raised it. + /// + /// A run of consecutive reads failed for lack of a connection. Detail is the number of failures that raised it. + /// ConnectionLost, /// A constraint refused a written value (23514). Detail is the pen's name. diff --git a/SemiPlot/SemiPlot.Core/Data/IDataProvider.cs b/SemiPlot/SemiPlot.Core/Data/IDataProvider.cs index cfd6f92..94106ce 100644 --- a/SemiPlot/SemiPlot.Core/Data/IDataProvider.cs +++ b/SemiPlot/SemiPlot.Core/Data/IDataProvider.cs @@ -9,11 +9,6 @@ public interface IDataProvider // Cold per call: no samples flow until subscribed; the subscriber disposes the returned IDisposable. IObservable> Subscribe(IReadOnlyList penIds); - // Hot, shared and never terminating: a consumer subscribes with an onNext handler alone. Every - // subscription's first successful tick reports Connected, the only point it is known armed, and a - // run of failed ticks reports the fault here rather than through OnError. - IObservable ConnectionFaults { get; } - Task>> QueryPensAsync(); Task>> QueryHistoryAsync( diff --git a/SemiPlot/SemiPlot.DataSource.Postgres/ArchiveExceptionMapper.cs b/SemiPlot/SemiPlot.DataSource.Postgres/ArchiveExceptionMapper.cs index 1866308..a6ef558 100644 --- a/SemiPlot/SemiPlot.DataSource.Postgres/ArchiveExceptionMapper.cs +++ b/SemiPlot/SemiPlot.DataSource.Postgres/ArchiveExceptionMapper.cs @@ -86,8 +86,23 @@ private ArchiveError Classify(Exception exception, string? relation) return Fault(IsConnectionFailure(exception) ? ArchiveFault.Unreachable : ArchiveFault.ReadFailed); } + // docs/architecture/data-integration.md#the-connection-state-the-catalogue-read-reports + private static bool EndsOrRefusesSession(string sqlState) + { + return sqlState.StartsWith("08", StringComparison.Ordinal) + || sqlState is PostgresErrorCodes.AdminShutdown + or PostgresErrorCodes.CrashShutdown + or PostgresErrorCodes.CannotConnectNow + or PostgresErrorCodes.TooManyConnections; + } + private ArchiveError MapSqlState(PostgresException postgres, string? relation) { + if (EndsOrRefusesSession(postgres.SqlState)) + { + return Fault(ArchiveFault.Unreachable); + } + return postgres.SqlState switch { PostgresErrorCodes.InvalidCatalogName => Fault(ArchiveFault.DatabaseMissing), diff --git a/SemiPlot/SemiPlot.DataSource.Postgres/Configuration/PostgresConnectionSettings.cs b/SemiPlot/SemiPlot.DataSource.Postgres/Configuration/PostgresConnectionSettings.cs index 39e841a..039095b 100644 --- a/SemiPlot/SemiPlot.DataSource.Postgres/Configuration/PostgresConnectionSettings.cs +++ b/SemiPlot/SemiPlot.DataSource.Postgres/Configuration/PostgresConnectionSettings.cs @@ -23,7 +23,7 @@ public sealed record PostgresConnectionSettings( public const int ConnectTimeoutSeconds = 15; /// - /// Client backstop for a server that stops answering; the poll overrides it per tick. + /// Client backstop for a server that stops answering; the poll's tick and the catalogue read override it. /// public const int CommandTimeoutSeconds = 300; diff --git a/SemiPlot/SemiPlot.DataSource.Postgres/PostgresDataProvider.cs b/SemiPlot/SemiPlot.DataSource.Postgres/PostgresDataProvider.cs index cb6625c..9588c47 100644 --- a/SemiPlot/SemiPlot.DataSource.Postgres/PostgresDataProvider.cs +++ b/SemiPlot/SemiPlot.DataSource.Postgres/PostgresDataProvider.cs @@ -3,7 +3,6 @@ using System.Globalization; using System.Reactive.Concurrency; using System.Reactive.Linq; -using System.Reactive.Subjects; using FluentResults; @@ -24,7 +23,7 @@ namespace SemiPlot.DataSource.Postgres; /// configured variables, the span they cover, /// a window of one layer and the live edge, all crossing the boundary in UTC. /// -public sealed class PostgresDataProvider : IDataProvider, IDisposable +public sealed class PostgresDataProvider : IDataProvider { /// /// The most pen groups one history read runs concurrently @@ -32,6 +31,9 @@ public sealed class PostgresDataProvider : IDataProvider, IDisposable /// internal const int HistoryReadConcurrency = 4; + // docs/architecture/data-integration.md#configuration + private const int CatalogueCommandTimeoutSeconds = 5; + private const string UncommissionedPenColor = "#808080"; private const string HistoryReadRequestTemplate = "History read at {Layer} from {FromUtc:O} to {ToUtc:O}, {TargetColumnCount} columns for {PenCount} pens " @@ -46,19 +48,11 @@ public sealed class PostgresDataProvider : IDataProvider, IDisposable private readonly ILogger _logger; private readonly int _historyReadConcurrency; - private readonly Subject _connectionFaultsSource = new(); - - // PollAsync's OnNext and Dispose's OnCompleted run on different threads; Synchronize() serializes them, - // as System.Reactive's contract for a multi-writer Subject requires. - private readonly ISubject _connectionFaults; - // docs/architecture/data-integration.md#the-catalogue-while-the-viewer-runs // Written from whichever pool thread a catalogue read resumes on. private readonly ConcurrentDictionary<(int PenId, int Column, string? StoredValue), byte> _warnedNormalisations = new(); - private int _disposed; - internal PostgresDataProvider( NpgsqlDataSource dataSource, ArchiveTimeConverter timeConverter, @@ -77,15 +71,8 @@ internal PostgresDataProvider( _timeProvider = timeProvider; _logger = logger; _historyReadConcurrency = historyReadConcurrency; - _connectionFaults = Subject.Synchronize(_connectionFaultsSource); } - /// - /// The connection state of the live edge. Hot, shared by every subscription and never terminating, as - /// states. - /// - public IObservable ConnectionFaults => _connectionFaults; - /// /// The live edge of the requested variables. Cold: each subscription starts a poll loop of its own, on the /// injected scheduler and at the operator's , holding @@ -107,7 +94,7 @@ public async Task>> QueryPensAsync() try { await using var connection = await _dataSource.OpenConnectionAsync().ConfigureAwait(false); - await using var command = new NpgsqlCommand(ArchiveStatements.PenCatalog, connection); + await using var command = CreateCatalogueCommand(connection); await using var reader = await command.ExecuteReaderAsync().ConfigureAwait(false); var pens = new List(); @@ -243,22 +230,6 @@ public async Task>> QueryArchiveDaysAsync( } } - /// - /// Completes ; the container disposes the singleton provider. - /// - public void Dispose() - { - if (Interlocked.Exchange(ref _disposed, 1) != 0) - { - return; - } - - // The raw subject is never disposed: PollAsync keeps writing OnNext after Dispose returns (its - // loop is not cancelled here), and OnNext on a disposed Subject throws where OnNext after - // OnCompleted is a silent no-op. Subject holds no unmanaged resource, so completing it is enough. - _connectionFaults.OnCompleted(); - } - // docs/architecture/data-integration.md#operation-to-statement private async Task> ReadHistoryAsync( int[] penIds, @@ -375,7 +346,6 @@ private async Task PollAsync( _dataSource, _timeConverter, _exceptionMapper, - _settings, penIds, _logger); @@ -383,7 +353,7 @@ private async Task PollAsync( { while (!cancellationToken.IsCancellationRequested) { - var tick = await poll.ReadOnceAsync(cancellationToken).ConfigureAwait(false); + var samples = await poll.ReadOnceAsync(cancellationToken).ConfigureAwait(false); // A disposal landing while the query is in flight delivers nothing further: the subscriber // is gone, and a batch arriving behind its back is a leaked loop by another name. @@ -392,14 +362,9 @@ private async Task PollAsync( return; } - if (tick.Samples.Count > 0) + if (samples.Count > 0) { - observer.OnNext(tick.Samples); - } - - if (tick.StateChange is { } state) - { - _connectionFaults.OnNext(state); + observer.OnNext(samples); } await _scheduler.Sleep(_settings.PollInterval, cancellationToken).ConfigureAwait(false); @@ -411,6 +376,14 @@ private async Task PollAsync( } } + internal static NpgsqlCommand CreateCatalogueCommand(NpgsqlConnection connection) + { + return new NpgsqlCommand(ArchiveStatements.PenCatalog, connection) + { + CommandTimeout = CatalogueCommandTimeoutSeconds + }; + } + // Bound on the archive's own wall clock: a UTC round trip is not injective across DST. internal static void BindLocalBucketedWindow( NpgsqlCommand command, diff --git a/SemiPlot/SemiPlot.DataSource.Postgres/RealtimePoll.cs b/SemiPlot/SemiPlot.DataSource.Postgres/RealtimePoll.cs index a612c5a..b77b7ba 100644 --- a/SemiPlot/SemiPlot.DataSource.Postgres/RealtimePoll.cs +++ b/SemiPlot/SemiPlot.DataSource.Postgres/RealtimePoll.cs @@ -1,24 +1,13 @@ -using System.Globalization; - using Microsoft.Extensions.Logging; using Npgsql; using NpgsqlTypes; -using SemiPlot.Core.Data; -using SemiPlot.Core.Data.Errors; using SemiPlot.Core.Trends; -using SemiPlot.DataSource.Postgres.Configuration; namespace SemiPlot.DataSource.Postgres; -/// -/// One tick's result. is null when the tick changed nothing the operator has to -/// be told about; otherwise it is the connection state the subscription has just moved to. -/// -internal readonly record struct RealtimeTick(IReadOnlyList Samples, ArchiveConnectionState? StateChange); - /// /// The live edge of one subscription, as an awaitable read rather than as an observable; one instance per /// subscription, and not thread-safe, the caller drives it from one loop at a time. @@ -27,15 +16,13 @@ internal sealed class RealtimePoll( NpgsqlDataSource dataSource, ArchiveTimeConverter timeConverter, ArchiveExceptionMapper exceptionMapper, - PostgresConnectionSettings settings, int[] penIds, ILogger logger) { // docs/architecture/scada-archive.md#quality-and-gaps private const int LastBeforeBreakQuality = 32; - // Must stay far below the connection string's backstop: three stalled ticks must raise the fault - // inside half a minute. + // A stalled tick holds the live edge for at most this long, not for the connection string's backstop. private const int TickCommandTimeoutSeconds = 10; // docs/architecture/data-integration.md#realtime @@ -45,19 +32,15 @@ internal sealed class RealtimePoll( private int _consecutiveFailures; - private bool _faultRaised; - - private bool _reportedConnected; - // The archive's own naive wall clock, null until the baseline is read; never taken from the local clock: // a clock difference between the two hosts would drop or repeat the first seconds. private DateTime? _lastSeen; /// /// One tick. The first one establishes the baseline and emits no sample; every later one emits the - /// rows written since the previous tick, converted to UTC. + /// rows written since the previous tick, converted to UTC. A failed tick emits no sample. /// - public async Task ReadOnceAsync(CancellationToken cancellationToken) + public async Task> ReadOnceAsync(CancellationToken cancellationToken) { try { @@ -69,16 +52,20 @@ public async Task ReadOnceAsync(CancellationToken cancellationToke ? await ReadSamplesAsync(connection, lastSeen, cancellationToken).ConfigureAwait(false) : await ReadBaselineAsync(connection, cancellationToken).ConfigureAwait(false); - return new RealtimeTick(samples, Succeed()); + _consecutiveFailures = 0; + + return samples; } catch (OperationCanceledException) { - // Ahead of the general catch: a self-cancelled read must not count towards the fault threshold. + // Ahead of the general catch: a self-cancelled read is no failed tick. throw; } catch (Exception exception) { - return new RealtimeTick(_noSamples, Fail(exception)); + LogFailure(exception); + + return _noSamples; } } @@ -126,12 +113,13 @@ private async Task> ReadBaselineAsync( { Advance(baseline); - logger.LogDebug("The realtime baseline for {PenCount} variables is {Baseline:O}.", penIds.Length, - baseline); + logger.LogDebug("The realtime baseline for {SubscribedPenCount} variables is {Baseline:O}.", + penIds.Length, baseline); } else { - logger.LogDebug("The {PenCount} subscribed variables carry no archived row yet.", penIds.Length); + logger.LogDebug("The {SubscribedPenCount} subscribed variables carry no archived row yet.", + penIds.Length); } return _noSamples; @@ -207,22 +195,7 @@ private void Advance(DateTime archiveLocal) } } - private ArchiveConnectionState? Succeed() - { - _consecutiveFailures = 0; - - if (_reportedConnected && !_faultRaised) - { - return null; - } - - _reportedConnected = true; - _faultRaised = false; - - return ArchiveConnectionState.Connected; - } - - private ArchiveConnectionState? Fail(Exception exception) + private void LogFailure(Exception exception) { var error = exceptionMapper.Map(exception, ArchiveStatements.TrendsRelation); @@ -233,19 +206,5 @@ private void Advance(DateTime archiveLocal) "The realtime poll tick failed, {ConsecutiveFailures} in a row: {Reason}", _consecutiveFailures, error.Message); - - if (_consecutiveFailures < ArchiveConnectionState.ConsecutiveFailuresBeforeFault || _faultRaised) - { - return null; - } - - _faultRaised = true; - - return new ArchiveConnectionState(new ArchiveError( - ArchiveFault.ConnectionLost, - settings.Host, - settings.Port, - settings.Database, - ArchiveConnectionState.ConsecutiveFailuresBeforeFault.ToString(CultureInfo.InvariantCulture))); } } diff --git a/SemiPlot/SemiPlot.Tests.Integration/ArchiveProviderFactory.cs b/SemiPlot/SemiPlot.Tests.Integration/ArchiveProviderFactory.cs index 4e87e2d..e75d459 100644 --- a/SemiPlot/SemiPlot.Tests.Integration/ArchiveProviderFactory.cs +++ b/SemiPlot/SemiPlot.Tests.Integration/ArchiveProviderFactory.cs @@ -35,14 +35,7 @@ public static ServiceProvider Build( { var services = new ServiceCollection(); - services.AddLogging(builder => - { - if (loggerProvider is not null) - { - builder.SetMinimumLevel(LogLevel.Debug); - builder.AddProvider(loggerProvider); - } - }); + services.AddTestLogging(loggerProvider); if (timeProvider is not null) { @@ -66,6 +59,19 @@ public static ServiceProvider Build( return services.BuildServiceProvider(); } + /// Sends every log line from Debug up to , when one is passed. + internal static IServiceCollection AddTestLogging(this IServiceCollection services, ILoggerProvider? loggerProvider) + { + return services.AddLogging(builder => + { + if (loggerProvider is not null) + { + builder.SetMinimumLevel(LogLevel.Debug); + builder.AddProvider(loggerProvider); + } + }); + } + private static PostgresConnectionSettings SettingsFor(string connectionString) { var builder = new NpgsqlConnectionStringBuilder(connectionString); diff --git a/SemiPlot/SemiPlot.Tests.Integration/ArmedGate.cs b/SemiPlot/SemiPlot.Tests.Integration/ArmedGate.cs index b981f9a..eeffec2 100644 --- a/SemiPlot/SemiPlot.Tests.Integration/ArmedGate.cs +++ b/SemiPlot/SemiPlot.Tests.Integration/ArmedGate.cs @@ -1,76 +1,143 @@ -using SemiPlot.Core.Data; +using Microsoft.Extensions.Logging; namespace SemiPlot.Tests.Integration; -/// The nth Connected is the nth subscription's armed point; one caller awaits at a time. -internal sealed class ArmedGate : IDisposable +/// +/// Counts, across every logger of the container, the poll's baseline lines (the nth is the nth armed point) and +/// the highest run of failed ticks it logs; one caller awaits each count at a time. +/// +internal sealed class ArmedGate(ILoggerProvider? inner = null) : ILoggerProvider { - private readonly List _states = []; + public const string BaselineKey = "SubscribedPenCount"; + + public const string FailureRunKey = "ConsecutiveFailures"; private readonly Lock _guard = new(); - private readonly IDisposable _subscription; + private int _baselineLineCount; - private (int Count, TaskCompletionSource Reached)? _waiter; + private int _highestConsecutiveFailures; - private int _connectedCount; + private (int Count, TaskCompletionSource Reached)? _armedWaiter; - public ArmedGate(IObservable states) - { - _subscription = states.Subscribe(Record); - } + private (int Count, TaskCompletionSource Reached)? _failureWaiter; - public bool NoFaultWasSeen + public int HighestConsecutiveFailures { get { lock (_guard) { - return _states.All(state => state.IsConnected); + return _highestConsecutiveFailures; } } } - public Task Armed(int subscriptionCount) + public Task Armed(int baselineLineCount) { lock (_guard) { - if (_connectedCount >= subscriptionCount) + if (_baselineLineCount >= baselineLineCount) { return Task.CompletedTask; } var waiter = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); - _waiter = (subscriptionCount, waiter); + _armedWaiter = (baselineLineCount, waiter); return waiter.Task; } } + public Task FailuresReached(int consecutiveFailures) + { + lock (_guard) + { + if (_highestConsecutiveFailures >= consecutiveFailures) + { + return Task.CompletedTask; + } + + var waiter = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + + _failureWaiter = (consecutiveFailures, waiter); + + return waiter.Task; + } + } + + public ILogger CreateLogger(string categoryName) + { + return new ForwardingLogger(this, inner?.CreateLogger(categoryName)); + } + + // The inner provider belongs to the caller that passed it in. public void Dispose() { - _subscription.Dispose(); } - private void Record(ArchiveConnectionState state) + private void Record(TState state) { - lock (_guard) + if (state is not IReadOnlyList> values) { - _states.Add(state); + return; + } - if (!state.IsConnected) + lock (_guard) + { + foreach (var (key, value) in values) { - return; + if (key == BaselineKey) + { + _baselineLineCount++; + _armedWaiter = Release(_armedWaiter, _baselineLineCount); + } + else if (key == FailureRunKey && value is int consecutiveFailures) + { + _highestConsecutiveFailures = Math.Max(_highestConsecutiveFailures, consecutiveFailures); + _failureWaiter = Release(_failureWaiter, _highestConsecutiveFailures); + } } + } + } - _connectedCount++; + private static (int Count, TaskCompletionSource Reached)? Release( + (int Count, TaskCompletionSource Reached)? waiter, + int reached) + { + if (waiter is not { } pending || pending.Count > reached) + { + return waiter; + } - if (_waiter is { } waiter && waiter.Count <= _connectedCount) - { - waiter.Reached.TrySetResult(); - _waiter = null; - } + pending.Reached.TrySetResult(); + + return null; + } + + private sealed class ForwardingLogger(ArmedGate gate, ILogger? inner) : ILogger + { + public IDisposable? BeginScope(TState state) + where TState : notnull + { + return inner?.BeginScope(state); + } + + public bool IsEnabled(LogLevel logLevel) + { + return true; + } + + public void Log( + LogLevel logLevel, + EventId eventId, + TState state, + Exception? exception, + Func formatter) + { + gate.Record(state); + inner?.Log(logLevel, eventId, state, exception, formatter); } } } diff --git a/SemiPlot/SemiPlot.Tests.Integration/CatalogueReadUnreachableTests.cs b/SemiPlot/SemiPlot.Tests.Integration/CatalogueReadUnreachableTests.cs new file mode 100644 index 0000000..7acedb0 --- /dev/null +++ b/SemiPlot/SemiPlot.Tests.Integration/CatalogueReadUnreachableTests.cs @@ -0,0 +1,34 @@ +using AwesomeAssertions; + +using Microsoft.Extensions.DependencyInjection; + +using SemiPlot.Core.Data; +using SemiPlot.Core.Data.Errors; +using SemiPlot.DataSource.Postgres.Configuration; + +using Xunit; + +namespace SemiPlot.Tests.Integration; + +[Trait("Component", "Core")] +[Trait("Area", "Data")] +[Trait("Category", "Integration")] +public sealed class CatalogueReadUnreachableTests +{ + private static readonly PostgresConnectionSettings _settings = UnreachableArchive.Settings(TimeSpan.FromSeconds(1)); + + [Fact] + public async Task ACatalogueReadOverARefusedSocket_AnswersUnreachableNamingTheArchive() + { + await using var services = UnreachableArchive.Services(_settings); + + var read = await services.GetRequiredService().QueryPensAsync(); + + read.IsFailed.Should().BeTrue(); + var error = read.Errors.Should().ContainSingle().Which.Should().BeOfType().Which; + error.Kind.Should().Be(ArchiveFault.Unreachable); + error.Host.Should().Be(_settings.Host); + error.Port.Should().Be(_settings.Port); + error.Database.Should().Be(_settings.Database); + } +} diff --git a/SemiPlot/SemiPlot.Tests.Integration/Journeys/LiveEdgeArchiveJourneyTests.cs b/SemiPlot/SemiPlot.Tests.Integration/Journeys/LiveEdgeArchiveJourneyTests.cs index 9b78397..fa8f27d 100644 --- a/SemiPlot/SemiPlot.Tests.Integration/Journeys/LiveEdgeArchiveJourneyTests.cs +++ b/SemiPlot/SemiPlot.Tests.Integration/Journeys/LiveEdgeArchiveJourneyTests.cs @@ -162,7 +162,8 @@ public async Task RowsOnAVariableOfTheirOwnReachTheChartWithoutBreakingAnyPen() // into the last column instead, and the assertions read the appended point. private async Task OpenStandAsync() { - var services = ArchiveProviderFactory.Build(Database.PlotConnectionString); + var armed = new ArmedGate(); + var services = ArchiveProviderFactory.Build(Database.PlotConnectionString, armed); LiveEdgeStand? stand = null; try @@ -178,10 +179,6 @@ private async Task OpenStandAsync() var coordinator = new TrendCoordinator( dataProvider, catalogue.Value, dataScheduler, ImmediateScheduler.Instance); - // Before the view model, because its constructor takes the first RefCount subscription on - // RealtimeBatches and that is what starts the poll. A gate opened after the poll's first tick would - // have nothing left to wait for and would never complete. - var armed = new ArmedGate(coordinator.ConnectionFaults); var messagePanel = new MessagePanelViewModel(); var chart = new TrendChartViewModel( coordinator, @@ -217,6 +214,7 @@ private async Task OpenStandAsync() if (stand is null) { await services.DisposeAsync(); + armed.Dispose(); } else { @@ -310,10 +308,10 @@ public async ValueTask DisposeAsync() batches.Dispose(); chart.Dispose(); messagePanel.Dispose(); - armed.Dispose(); coordinator.Dispose(); await services.DisposeAsync(); + armed.Dispose(); } } } diff --git a/SemiPlot/SemiPlot.Tests.Integration/RealtimeArchive.cs b/SemiPlot/SemiPlot.Tests.Integration/RealtimeArchive.cs index 19cddec..3922bb9 100644 --- a/SemiPlot/SemiPlot.Tests.Integration/RealtimeArchive.cs +++ b/SemiPlot/SemiPlot.Tests.Integration/RealtimeArchive.cs @@ -6,13 +6,9 @@ using NpgsqlTypes; -using SemiPlot.Core.Data; using SemiPlot.DataSource.Postgres; -using SemiPlot.DataSource.Postgres.Configuration; using SemiPlot.Tools.ArchiveSeeder; -using Xunit; - namespace SemiPlot.Tests.Integration; /// The archive rows, the subscription and the append path the realtime test classes share. @@ -22,10 +18,6 @@ internal static class RealtimeArchive INSERT INTO public.trends (id, l, t, v, q) VALUES (@id, 0, @t, @v, @q); """; - // Restated here rather than read off RealtimePoll: the threshold is a promise to the operator, and a - // test computing it from the constant it guards would pass whatever the constant became. - public const int ConsecutiveFailuresBeforeFault = 3; - // A strict subset of the seeded identifiers, so a tick that ignored @ids would fail. public static readonly int[] SubscribedPenIds = [1, 2]; @@ -45,23 +37,10 @@ public static RealtimePoll NewPoll(IServiceProvider services, ILogger? logger = services.GetRequiredService(), services.GetRequiredService(), services.GetRequiredService(), - services.GetRequiredService(), SubscribedPenIds, logger ?? NullLogger.Instance); } - public static async Task> TickAsync(RealtimePoll poll, int tickCount) - { - var states = new List(); - - for (var tick = 0; tick < tickCount; tick++) - { - states.Add((await poll.ReadOnceAsync(TestContext.Current.CancellationToken)).StateChange); - } - - return states; - } - // Two rows per subscribed variable, the second of them the baseline every subscription starts from. public static IReadOnlyList SeededRows() { diff --git a/SemiPlot/SemiPlot.Tests.Integration/RealtimeEmptyArchiveTests.cs b/SemiPlot/SemiPlot.Tests.Integration/RealtimeEmptyArchiveTests.cs index d312bb7..57614e9 100644 --- a/SemiPlot/SemiPlot.Tests.Integration/RealtimeEmptyArchiveTests.cs +++ b/SemiPlot/SemiPlot.Tests.Integration/RealtimeEmptyArchiveTests.cs @@ -1,6 +1,7 @@ using AwesomeAssertions; using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging.Testing; using SemiPlot.Core.Data; @@ -11,7 +12,7 @@ namespace SemiPlot.Tests.Integration; // The branch every other gated class seeds past: an archive with public.trends but no row for the -// subscribed variables, whose baseline scalar answers NULL and must still arm the subscription. +// subscribed variables, whose baseline scalar answers NULL. [Collection(ArchiveDatabaseCollection.Name)] [Trait("Component", "Core")] [Trait("Area", "Data")] @@ -20,25 +21,23 @@ public sealed class RealtimeEmptyArchiveTests(PostgresContainerFixture postgresC : ClonedArchiveTest(postgresContainerFixture, CloneSource.Provisioned) { // A NULL answer leaves lastSeen unset, so the next tick repeats the baseline read rather than binding - // @lastSeen to nothing and going blind for good. The first tick still arms the subscription. + // @lastSeen to nothing and going blind for good. [Fact] - public async Task TheBaselineBranchArmsTheSubscriptionAndRepeatsUntilTheArchiveCarriesARow() + public async Task TheBaselineBranchRepeatsUntilTheArchiveCarriesARow() { var cancellationToken = TestContext.Current.CancellationToken; using var services = ArchiveProviderFactory.Build(Database.PlotConnectionString); + using var logs = new FakeLoggerProvider(); - var poll = NewPoll(services); - - var first = await poll.ReadOnceAsync(cancellationToken); + var poll = NewPoll(services, logs.CreateLogger("RealtimePoll")); - first.Samples.Should().BeEmpty(); - first.StateChange.Should().BeSameAs(ArchiveConnectionState.Connected); + (await poll.ReadOnceAsync(cancellationToken)).Should().BeEmpty(); + (await poll.ReadOnceAsync(cancellationToken)).Should().BeEmpty(); - var second = await poll.ReadOnceAsync(cancellationToken); - - second.Samples.Should().BeEmpty(); - second.StateChange.Should().BeNull(); + logs.Collector.GetSnapshot() + .Count(entry => entry.GetStructuredStateValue(ArmedGate.BaselineKey) is not null) + .Should().Be(2); } // A tick that emitted the first row ever written would have to emit every row since the archive began. @@ -58,26 +57,24 @@ public async Task TheFirstEverRowBecomesTheBaselineAndTheNextOneIsDelivered() var baseline = await poll.ReadOnceAsync(cancellationToken); - baseline.Samples.Should().BeEmpty(); + baseline.Should().BeEmpty(); await AppendAsync(Database, SubscribedPenIds[1], second, 2.5, cancellationToken: cancellationToken); - var sample = (await poll.ReadOnceAsync(cancellationToken)).Samples.Should().ContainSingle().Which; + var sample = (await poll.ReadOnceAsync(cancellationToken)).Should().ContainSingle().Which; sample.PenId.Should().Be(SubscribedPenIds[1]); sample.TimestampUtc.Should().Be(ArchiveProviderFactory.TimeConverter.ToUtc(second)); } - // The same branch through the composed provider, which is where a consumer meets it: the armed signal - // has to reach ConnectionFaults over an archive that answers nothing. [Fact] - public async Task ASubscriptionOverAnArchiveWithNoRowReportsConnected() + public async Task ASubscriptionOverAnArchiveWithNoRowLogsItsBaseline() { - using var services = ArchiveProviderFactory.Build(Database.PlotConnectionString); + using var armed = new ArmedGate(); + using var services = ArchiveProviderFactory.Build(Database.PlotConnectionString, armed); var provider = services.GetRequiredService(); - using var armed = new ArmedGate(provider.ConnectionFaults); using var subscription = provider.Subscribe(SubscribedPenIds).Subscribe(_ => { }); await armed.Armed(1).WaitAsync(GateBound, TestContext.Current.CancellationToken); diff --git a/SemiPlot/SemiPlot.Tests.Integration/RealtimePollReadTests.cs b/SemiPlot/SemiPlot.Tests.Integration/RealtimePollReadTests.cs index 3a94868..1777280 100644 --- a/SemiPlot/SemiPlot.Tests.Integration/RealtimePollReadTests.cs +++ b/SemiPlot/SemiPlot.Tests.Integration/RealtimePollReadTests.cs @@ -2,8 +2,6 @@ using Microsoft.Extensions.Logging.Testing; -using SemiPlot.Core.Data; -using SemiPlot.Core.Data.Errors; using SemiPlot.Core.Trends; using SemiPlot.Tools.ArchiveSeeder; @@ -36,14 +34,13 @@ protected override async ValueTask SeedAsync() // It emits nothing: a first tick that emitted rows would have to emit every row since the archive began. [Fact] - public async Task TheFirstTickEmitsNoSampleAndReportsTheSubscriptionArmed() + public async Task TheFirstTickEmitsNoSample() { using var services = ArchiveProviderFactory.Build(Database.PlotConnectionString); - var tick = await NewPoll(services).ReadOnceAsync(TestContext.Current.CancellationToken); + var samples = await NewPoll(services).ReadOnceAsync(TestContext.Current.CancellationToken); - tick.Samples.Should().BeEmpty(); - tick.StateChange.Should().BeSameAs(ArchiveConnectionState.Connected); + samples.Should().BeEmpty(); } [Fact] @@ -65,16 +62,13 @@ await AppendAsync( ArchiveRow.OrdinaryQuality, cancellationToken: cancellationToken); - var tick = await poll.ReadOnceAsync(cancellationToken); + var samples = await poll.ReadOnceAsync(cancellationToken); - var sample = tick.Samples.Should().ContainSingle().Which; + var sample = samples.Should().ContainSingle().Which; sample.PenId.Should().Be(SubscribedPenIds[0]); sample.Value.Should().Be(42.5); sample.TimestampUtc.Should().Be(ArchiveProviderFactory.TimeConverter.ToUtc(appended)); - - // The subscription was armed by the first tick, so an ordinary delivery reports no state change. - tick.StateChange.Should().BeNull(); } [Fact] @@ -88,10 +82,9 @@ public async Task ATickWithNothingNewEmitsNothing() await poll.ReadOnceAsync(cancellationToken); - var tick = await poll.ReadOnceAsync(cancellationToken); + var samples = await poll.ReadOnceAsync(cancellationToken); - tick.Samples.Should().BeEmpty(); - tick.StateChange.Should().BeNull(); + samples.Should().BeEmpty(); } [Fact] @@ -112,15 +105,15 @@ await AppendAsync( ArchiveRow.OrdinaryQuality, cancellationToken: cancellationToken); - var tick = await poll.ReadOnceAsync(cancellationToken); + var samples = await poll.ReadOnceAsync(cancellationToken); - tick.Samples.Should().BeEmpty(); + samples.Should().BeEmpty(); } // The poll advances past the null row, or it would re-read the same row on every later tick. A delivered // row cannot show that, so the tick after the null row reads its own debug entry: it must count no row. [Fact] - public async Task ARowCarryingANullValueEmitsNoSampleReportsNoFaultAndIsNotReadAgain() + public async Task ARowCarryingANullValueEmitsNoSampleAndIsNotReadAgain() { var cancellationToken = TestContext.Current.CancellationToken; @@ -138,10 +131,9 @@ await AppendAsync( ArchiveRow.OrdinaryQuality, cancellationToken: cancellationToken); - var tick = await poll.ReadOnceAsync(cancellationToken); + var samples = await poll.ReadOnceAsync(cancellationToken); - tick.Samples.Should().BeEmpty(); - tick.StateChange.Should().BeNull(); + samples.Should().BeEmpty(); await poll.ReadOnceAsync(cancellationToken); @@ -169,9 +161,9 @@ await AppendAsync( ArchiveRow.LastBeforeBreakQuality, cancellationToken: cancellationToken); - var tick = await poll.ReadOnceAsync(cancellationToken); + var samples = await poll.ReadOnceAsync(cancellationToken); - tick.Samples.Should().Equal( + samples.Should().Equal( new Sample(SubscribedPenIds[0], appendedUtc, 11.25), new Sample(SubscribedPenIds[0], appendedUtc.AddMilliseconds(1), null)); PollCounts(logs).Should().Equal(("1", "1", "0", "1")); @@ -198,35 +190,37 @@ await AppendAsync( ArchiveRow.LastBeforeBreakQuality, cancellationToken: cancellationToken); - var tick = await poll.ReadOnceAsync(cancellationToken); + var samples = await poll.ReadOnceAsync(cancellationToken); - tick.Samples.Should().Equal(new Sample( + samples.Should().Equal(new Sample( SubscribedPenIds[0], ArchiveProviderFactory.TimeConverter.ToUtc(appended).AddMilliseconds(1), null)); - tick.StateChange.Should().BeNull(); await poll.ReadOnceAsync(cancellationToken); PollCounts(logs).Should().Equal(("1", "0", "1", "1"), ("0", "0", "0", "0")); } - // Renamed rather than dropped: the container is shared. The tick after recovery must report nothing. + // Renamed rather than dropped: the container is shared. [Fact] - public async Task TheFirstSuccessAfterARaisedFaultReportsConnectedExactlyOnce() + public async Task TheFirstTickAfterTheTableReturnsTakesTheBaseline() { + const int FailedTickCount = 3; var cancellationToken = TestContext.Current.CancellationToken; using var services = ArchiveProviderFactory.Build(Database.PlotConnectionString); var poll = NewPoll(services); - var states = new List(); await HideTrendsAsync(cancellationToken); try { - states.AddRange(await TickAsync(poll, ConsecutiveFailuresBeforeFault)); + for (var tick = 0; tick < FailedTickCount; tick++) + { + (await poll.ReadOnceAsync(cancellationToken)).Should().BeEmpty(); + } } finally { @@ -248,8 +242,7 @@ await AppendAsync( var recovery = await poll.ReadOnceAsync(cancellationToken); - states.Add(recovery.StateChange); - recovery.Samples.Should().BeEmpty(); + recovery.Should().BeEmpty(); await AppendAsync( Database, @@ -261,20 +254,10 @@ await AppendAsync( var delivery = await poll.ReadOnceAsync(cancellationToken); - states.Add(delivery.StateChange); - - var delivered = delivery.Samples.Should().ContainSingle().Which; + var delivered = delivery.Should().ContainSingle().Which; delivered.PenId.Should().Be(SubscribedPenIds[1]); delivered.TimestampUtc.Should().Be(ArchiveProviderFactory.TimeConverter.ToUtc(appendedSecond)); - - states.Take(ConsecutiveFailuresBeforeFault - 1).Should().AllSatisfy(state => state.Should().BeNull()); - - var raisedFault = states[ConsecutiveFailuresBeforeFault - 1]?.Fault?.Kind; - - raisedFault.Should().Be(ArchiveFault.ConnectionLost); - states[ConsecutiveFailuresBeforeFault].Should().BeSameAs(ArchiveConnectionState.Connected); - states[ConsecutiveFailuresBeforeFault + 1].Should().BeNull(); } // Renamed rather than dropped: the rows and the day partitions come back with it, so the recovering diff --git a/SemiPlot/SemiPlot.Tests.Integration/RealtimePollTests.cs b/SemiPlot/SemiPlot.Tests.Integration/RealtimePollTests.cs index f8d0023..94a3eab 100644 --- a/SemiPlot/SemiPlot.Tests.Integration/RealtimePollTests.cs +++ b/SemiPlot/SemiPlot.Tests.Integration/RealtimePollTests.cs @@ -1,6 +1,7 @@ using AwesomeAssertions; -using SemiPlot.Core.Data.Errors; +using Microsoft.Extensions.Logging.Testing; + using SemiPlot.DataSource.Postgres.Configuration; using Xunit; @@ -9,43 +10,45 @@ namespace SemiPlot.Tests.Integration; -// The failure ladder alone, which needs no server: every tick connects to an address nothing answers and -// fails there. What a successful tick reads is a database question and is covered by RealtimePollReadTests. +// A failed tick and its logged run of failures, which needs no server: every tick connects to an address +// nothing answers and fails there. What a successful tick reads is a database question and is covered by +// RealtimePollReadTests. [Trait("Component", "Core")] [Trait("Area", "Data")] [Trait("Category", "Integration")] public sealed class RealtimePollTests { + private const int TickCount = 5; + private static readonly PostgresConnectionSettings _settings = UnreachableArchive.Settings(TimeSpan.FromSeconds(1)); [Fact] - public async Task ASingleFailureReportsNoSampleAndRaisesNoFault() + public async Task AFailedTickEmitsNoSample() { await using var services = UnreachableArchive.Services(_settings); - var tick = await NewPoll(services).ReadOnceAsync(TestContext.Current.CancellationToken); + var samples = await NewPoll(services).ReadOnceAsync(TestContext.Current.CancellationToken); - tick.Samples.Should().BeEmpty(); - tick.StateChange.Should().BeNull(); + samples.Should().BeEmpty(); } [Fact] - public async Task TheThirdConsecutiveFailureRaisesExactlyOneFaultNamingTheArchive() + public async Task EachFailedTickLogsTheRunOfFailuresItExtends() { - await using var services = UnreachableArchive.Services(_settings); + var cancellationToken = TestContext.Current.CancellationToken; - var states = await TickAsync(NewPoll(services), ConsecutiveFailuresBeforeFault + 2); + await using var services = UnreachableArchive.Services(_settings); + using var logs = new FakeLoggerProvider(); - states.Take(ConsecutiveFailuresBeforeFault - 1).Should().AllSatisfy(state => state.Should().BeNull()); + var poll = NewPoll(services, logs.CreateLogger("RealtimePoll")); - var thirdFault = states[ConsecutiveFailuresBeforeFault - 1]?.Fault; - var fault = thirdFault.Should().BeOfType().Which; + for (var tick = 0; tick < TickCount; tick++) + { + await poll.ReadOnceAsync(cancellationToken); + } - fault.Kind.Should().Be(ArchiveFault.ConnectionLost); - fault.Detail.Should().Be($"{ConsecutiveFailuresBeforeFault}"); - fault.Host.Should().Be(_settings.Host); - fault.Port.Should().Be(_settings.Port); - fault.Database.Should().Be(_settings.Database); - states.Skip(ConsecutiveFailuresBeforeFault).Should().AllSatisfy(state => state.Should().BeNull()); + logs.Collector.GetSnapshot() + .Select(entry => entry.GetStructuredStateValue(ArmedGate.FailureRunKey)) + .Should().Equal("1", "2", "3", "4", "5"); } } diff --git a/SemiPlot/SemiPlot.Tests.Integration/RealtimeSubscriptionDisposalTests.cs b/SemiPlot/SemiPlot.Tests.Integration/RealtimeSubscriptionDisposalTests.cs index de8a5d9..f6931ba 100644 --- a/SemiPlot/SemiPlot.Tests.Integration/RealtimeSubscriptionDisposalTests.cs +++ b/SemiPlot/SemiPlot.Tests.Integration/RealtimeSubscriptionDisposalTests.cs @@ -14,36 +14,37 @@ namespace SemiPlot.Tests.Integration; [Trait("Category", "Integration")] public sealed class RealtimeSubscriptionDisposalTests { + private const int CalibrationTicks = 3; + // Silence alone proves nothing (a leaked loop against this unreachable address is also silent), so a - // second, left-running provider's own ArchiveFault.ConnectionLost calibrates the wait. + // second, left-running provider's own failed ticks calibrate the wait. [Fact] public async Task SubscribingAndDisposingAtOnceDeliversNothingAndStopsThePoll() { var settings = UnreachableArchive.Settings(TimeSpan.FromMilliseconds(10)); - using var droppedServices = UnreachableArchive.Services(settings); - using var runningServices = UnreachableArchive.Services(settings); + using var droppedGate = new ArmedGate(); + using var runningGate = new ArmedGate(); + using var droppedServices = UnreachableArchive.Services(settings, droppedGate); + using var runningServices = UnreachableArchive.Services(settings, runningGate); var dropped = droppedServices.GetRequiredService(); var running = runningServices.GetRequiredService(); - var runningFaulted = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); - - using var droppedStates = new BatchCollector(dropped.ConnectionFaults); - using var runningWatch = running.ConnectionFaults.Subscribe(_ => runningFaulted.TrySetResult()); using var droppedBatches = new BatchCollector>(dropped.Subscribe([1])); droppedBatches.Dispose(); using var control = running.Subscribe([1]).Subscribe(_ => { }); - await runningFaulted.Task.WaitAsync(ClonedArchiveTest.GateBound, TestContext.Current.CancellationToken); + await runningGate.FailuresReached(CalibrationTicks) + .WaitAsync(ClonedArchiveTest.GateBound, TestContext.Current.CancellationToken); // The control needed three ticks to get here, so a loop that survived its own disposal has had the - // same three plus this margin. Anything it raised has been recorded by now. + // same three plus this margin. Anything it logged has been recorded by now. await Task.Delay(settings.PollInterval * 10, TestContext.Current.CancellationToken); droppedBatches.Batches.Should().BeEmpty(); - droppedStates.Batches.Should().BeEmpty(); + droppedGate.HighestConsecutiveFailures.Should().BeLessThan(CalibrationTicks); } } diff --git a/SemiPlot/SemiPlot.Tests.Integration/RealtimeSubscriptionTests.cs b/SemiPlot/SemiPlot.Tests.Integration/RealtimeSubscriptionTests.cs index bd3c8ed..3e53921 100644 --- a/SemiPlot/SemiPlot.Tests.Integration/RealtimeSubscriptionTests.cs +++ b/SemiPlot/SemiPlot.Tests.Integration/RealtimeSubscriptionTests.cs @@ -12,9 +12,9 @@ namespace SemiPlot.Tests.Integration; -// Every synchronisation point is an awaited emission (Connected, or a delivered batch) bounded by GateBound, -// except the disposal check, which waits out poll intervals because a stopped poll emits nothing. Appending -// before the baseline has run races the subscription and fails the await at the bound. +// Every synchronisation point is an awaited emission (the poll's baseline log line, or a delivered batch) +// bounded by GateBound, except the disposal check, which waits out poll intervals because a stopped poll emits +// nothing. Appending before the baseline has run races the subscription and fails the await at the bound. [Collection(ArchiveDatabaseCollection.Name)] [Trait("Component", "Core")] [Trait("Area", "Data")] @@ -22,6 +22,8 @@ namespace SemiPlot.Tests.Integration; public sealed class RealtimeSubscriptionTests(PostgresContainerFixture postgresContainerFixture) : ClonedArchiveTest(postgresContainerFixture, CloneSource.Provisioned) { + private const int FailureRunBound = 3; + protected override async ValueTask SeedAsync() { await Writer().WriteAsync(SeededRows(), Day, NextDay); @@ -36,11 +38,11 @@ public async Task DisposingASubscriptionStopsItsPoll() var cancellationToken = TestContext.Current.CancellationToken; using var logs = new FakeLoggerProvider(); - using var services = ArchiveProviderFactory.Build(Database.PlotConnectionString, logs); + using var armed = new ArmedGate(logs); + using var services = ArchiveProviderFactory.Build(Database.PlotConnectionString, armed); var provider = services.GetRequiredService(); - using var armed = new ArmedGate(provider.ConnectionFaults); using var subscription = new BatchCollector>(provider.Subscribe(SubscribedPenIds)); await armed.Armed(1).WaitAsync(GateBound, cancellationToken); @@ -77,11 +79,11 @@ public async Task TwoSubscriptionsEachKeepTheirOwnLastSeen() var beforeTheSecondSubscription = SeededLast.AddSeconds(1); var afterBoth = SeededLast.AddSeconds(2); - using var services = ArchiveProviderFactory.Build(Database.PlotConnectionString); + using var armed = new ArmedGate(); + using var services = ArchiveProviderFactory.Build(Database.PlotConnectionString, armed); var provider = services.GetRequiredService(); - using var armed = new ArmedGate(provider.ConnectionFaults); using var first = new BatchCollector>(provider.Subscribe(SubscribedPenIds)); await armed.Armed(1).WaitAsync(GateBound, cancellationToken); @@ -104,9 +106,9 @@ await AppendAsync( await firstSeesTheLateRow.WaitAsync(GateBound, cancellationToken); await secondSeesTheLateRow.WaitAsync(GateBound, cancellationToken); - armed.NoFaultWasSeen.Should().BeTrue( - "the second subscription's armed point is the second connected state only while no fault has " - + "been raised and cleared behind it"); + armed.HighestConsecutiveFailures.Should().BeLessThan( + FailureRunBound, + "the test relies on an archive that answers the poll, and a run of failed ticks this long says it did not"); Delivered(first).Should().Equal( [ ArchiveProviderFactory.TimeConverter.ToUtc(beforeTheSecondSubscription), diff --git a/SemiPlot/SemiPlot.Tests.Integration/UnreachableArchive.cs b/SemiPlot/SemiPlot.Tests.Integration/UnreachableArchive.cs index 4c0b728..ab7684c 100644 --- a/SemiPlot/SemiPlot.Tests.Integration/UnreachableArchive.cs +++ b/SemiPlot/SemiPlot.Tests.Integration/UnreachableArchive.cs @@ -1,4 +1,5 @@ using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; using SemiPlot.DataSource.Postgres; using SemiPlot.DataSource.Postgres.Configuration; @@ -22,10 +23,12 @@ public static PostgresConnectionSettings Settings(TimeSpan pollInterval) Schema: "public"); } - public static ServiceProvider Services(PostgresConnectionSettings settings) + public static ServiceProvider Services( + PostgresConnectionSettings settings, + ILoggerProvider? loggerProvider = null) { return new ServiceCollection() - .AddLogging() + .AddTestLogging(loggerProvider) .AddPostgresData(settings) .BuildServiceProvider(); } diff --git a/SemiPlot/SemiPlot.Tests.Unit/Postgres/ArchiveExceptionMapperTests.cs b/SemiPlot/SemiPlot.Tests.Unit/Postgres/ArchiveExceptionMapperTests.cs index 549984c..84bef41 100644 --- a/SemiPlot/SemiPlot.Tests.Unit/Postgres/ArchiveExceptionMapperTests.cs +++ b/SemiPlot/SemiPlot.Tests.Unit/Postgres/ArchiveExceptionMapperTests.cs @@ -44,6 +44,33 @@ public void TheCommandBoundFiringMapsToUnreachableAndNotToQueryTimedOut() Map(exception).Kind.Should().Be(ArchiveFault.Unreachable); } + [Theory] + [InlineData("08006", ArchiveFault.Unreachable)] + [InlineData("57P01", ArchiveFault.Unreachable)] + [InlineData("57P02", ArchiveFault.Unreachable)] + [InlineData("57P03", ArchiveFault.Unreachable)] + [InlineData("53300", ArchiveFault.Unreachable)] + [InlineData("57014", ArchiveFault.QueryTimedOut)] + public void ARefusedSessionMapsToUnreachable_AndAnEndedStatementStaysQueryTimedOut( + string sqlState, + ArchiveFault kind) + { + var exception = Postgres(sqlState); + + var error = Map(exception); + + error.Kind.Should().Be(kind); + error.Detail.Should().Be(string.Empty); + AssertEndpoint(error); + error.Reasons.OfType().Should().ContainSingle().Which.Exception.Should().BeSameAs(exception); + } + + [Fact] + public void AWriteThatMeetsAnAdminShutdownMapsToUnreachable() + { + MapWrite(Postgres("57P01"), "Chamber pressure").Kind.Should().Be(ArchiveFault.Unreachable); + } + [Fact] public void AMissingDatabaseMapsToDatabaseMissingAndNamesNoTable() { diff --git a/SemiPlot/SemiPlot.Tests.Unit/Postgres/ArchiveStatementTextTests.cs b/SemiPlot/SemiPlot.Tests.Unit/Postgres/ArchiveStatementTextTests.cs index 4d2b374..e2b8280 100644 --- a/SemiPlot/SemiPlot.Tests.Unit/Postgres/ArchiveStatementTextTests.cs +++ b/SemiPlot/SemiPlot.Tests.Unit/Postgres/ArchiveStatementTextTests.cs @@ -217,7 +217,7 @@ public void TheBaselineBinderNamesExactlyTheStatementsOwnParameters() } // A tick must not inherit the connection string's five-minute backstop, or a server that accepts and - // then answers nothing would freeze the chart for minutes before the fault threshold banner shows. + // then answers nothing would hold the live edge for minutes. [Theory] [InlineData(ArchiveStatements.RealtimeBaseline)] [InlineData(ArchiveStatements.RealtimePoll)] @@ -232,6 +232,21 @@ public void ATicksStatementCarriesItsOwnBoundAndNotTheDataSourceBackstop(string command.CommandTimeout.Should().Be(TickBoundSeconds); } + // Three failed catalogue reads raise the archive fault, so each must fail at its own bound rather than at + // the connection string's five-minute backstop. + [Fact] + public void TheCatalogueCommandCarriesItsOwnBoundAndNotTheDataSourceBackstop() + { + const int CatalogueBoundSeconds = 5; + + using var connection = new NpgsqlConnection(); + + using var command = PostgresDataProvider.CreateCatalogueCommand(connection); + + command.CommandText.Should().Be(ArchiveStatements.PenCatalog); + command.CommandTimeout.Should().Be(CatalogueBoundSeconds); + } + private static void AssertBinderNamesTheStatementsOwnParameters(NpgsqlCommand command, string statement) { var bound = command.Parameters diff --git a/SemiPlot/SemiPlot.Tests.Unit/Postgres/PostgresCompositionTests.cs b/SemiPlot/SemiPlot.Tests.Unit/Postgres/PostgresCompositionTests.cs index ba6aff2..2fa5dd4 100644 --- a/SemiPlot/SemiPlot.Tests.Unit/Postgres/PostgresCompositionTests.cs +++ b/SemiPlot/SemiPlot.Tests.Unit/Postgres/PostgresCompositionTests.cs @@ -131,31 +131,4 @@ public void AddPostgresDataResolvesTheSettingsItWasGiven() services.GetRequiredService().Should().BeSameAs(settings); } - - [Fact] - public void DisposingTheProviderCompletesTheConnectionFaultStream() - { - using var services = BuildProvider(); - var provider = (PostgresDataProvider)services.GetRequiredService(); - var completed = false; - - using var watch = provider.ConnectionFaults.Subscribe(_ => { }, () => completed = true); - - provider.Dispose(); - - completed.Should().BeTrue(); - } - - [Fact] - public void DisposingTwiceDoesNotThrow() - { - using var services = BuildProvider(); - var provider = (PostgresDataProvider)services.GetRequiredService(); - - provider.Dispose(); - - var act = provider.Dispose; - - act.Should().NotThrow(); - } } diff --git a/SemiPlot/SemiPlot.Tests.Unit/UI/Bridge/FakeDataProvider.cs b/SemiPlot/SemiPlot.Tests.Unit/UI/Bridge/FakeDataProvider.cs index 766a933..d85bea0 100644 --- a/SemiPlot/SemiPlot.Tests.Unit/UI/Bridge/FakeDataProvider.cs +++ b/SemiPlot/SemiPlot.Tests.Unit/UI/Bridge/FakeDataProvider.cs @@ -1,8 +1,6 @@ using System.Collections; using System.Reactive.Concurrency; -using System.Reactive.Disposables; using System.Reactive.Linq; -using System.Reactive.Subjects; using AwesomeAssertions; @@ -30,10 +28,6 @@ internal sealed class FakeDataProvider( private readonly IScheduler _scheduler = scheduler; - // Hot and never completed, like the real provider's. A test pushes into it through ReportConnectionState - // instead of waiting for a tick that this fake never runs. - private readonly Subject _connectionFaults = new(); - // A test waits for a query on another thread and then reads it, so the log is appended and copied whole. private readonly Lock _historyQueriesGate = new(); private readonly List _historyQueries = []; @@ -51,6 +45,8 @@ internal sealed class FakeDataProvider( // returns, so a consumer routing on error type is exercised rather than one reading a message. public bool FailPens { get; set; } + public ArchiveFault PensFailureKind { get; set; } = ArchiveFault.Unreachable; + public bool FailExtent { get; set; } // A catalogue read that throws instead of answering. The real provider answers every failure with a @@ -164,27 +160,6 @@ private IReadOnlyList HistoryQueries // has to survive. The real provider catches everything inside its poll, so only this fake reaches it. public Exception? RealtimeStreamFailure { get; set; } - public IObservable ConnectionFaults => Observable.Create(observer => - { - ConnectionFaultsObserverCount++; - var subscription = _connectionFaults.Subscribe(observer); - - return Disposable.Create(() => - { - subscription.Dispose(); - ConnectionFaultsObserverCount--; - }); - }); - - public int ConnectionFaultsObserverCount { get; private set; } - - // The seam a test drives the connection banner from: the fake runs no poll, so nothing else would ever - // push a state onto the stream. - public void ReportConnectionState(ArchiveConnectionState state) - { - _connectionFaults.OnNext(state); - } - /// The next tick follows the pen's value with a break 1 ms later, as the poll's q = 32 row. public void BreakRealtimePenOnce(int penId) { @@ -282,7 +257,7 @@ public Task>> QueryPensAsync() if (FailPens) { return Task.FromResult( - Result.Fail>(UnreachableArchive())); + Result.Fail>(BenchArchiveError(PensFailureKind))); } if (GatePens) diff --git a/SemiPlot/SemiPlot.Tests.Unit/UI/Bridge/PenCatalogueSyncTests.cs b/SemiPlot/SemiPlot.Tests.Unit/UI/Bridge/PenCatalogueSyncTests.cs index a939409..e570fa2 100644 --- a/SemiPlot/SemiPlot.Tests.Unit/UI/Bridge/PenCatalogueSyncTests.cs +++ b/SemiPlot/SemiPlot.Tests.Unit/UI/Bridge/PenCatalogueSyncTests.cs @@ -5,6 +5,8 @@ using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Reactive.Testing; +using SemiPlot.Core.Data; +using SemiPlot.Core.Data.Errors; using SemiPlot.Core.Trends; using SemiPlot.UI.Bridge; using SemiPlot.UI.Messages; @@ -21,12 +23,14 @@ namespace SemiPlot.Tests.Unit.UI.Bridge; public sealed class PenCatalogueSyncTests : IDisposable { private static readonly TimeSpan _tick = TimeSpan.FromTicks(1); + private static readonly ArchiveError _tableMissing = FakeDataProvider.BenchArchiveError(ArchiveFault.TableMissing); private readonly TestScheduler _scheduler = new(); private readonly FakeDataProvider _provider; private readonly MessagePanelViewModel _messagePanel = new(); private readonly PenCatalogueSync _sync; private readonly List _deltas = []; + private readonly List _states = []; public PenCatalogueSyncTests() { @@ -34,6 +38,7 @@ public PenCatalogueSyncTests() _sync = new PenCatalogueSync( _provider, _provider.Pens, _messagePanel, _scheduler, NullLogger.Instance); _sync.Deltas.Subscribe(_deltas.Add); + _sync.ConnectionStates.Subscribe(_states.Add); } public void Dispose() @@ -138,26 +143,122 @@ public void AChangedRead_EmitsItsDeltaOnce_AndBecomesTheBaseline() } [Fact] - public void TwoFailedReads_ReportNothing_TheThirdReportsOnce_AndTheNextSuccessCarriesEverythingSince() + public void TheFirstSuccess_EmitsConnectedOnce_AndLaterSuccessesEmitNothing() + { + StartAndReadNow(); + _states.Should().Equal(ArchiveConnectionState.Connected); + + Advance(ArchiveReadLoop.ReadInterval * 3); + + _provider.PensQueryCount.Should().Be(4); + _states.Should().ContainSingle("only the first success is news"); + } + + [Fact] + public void TwoUnreachableReads_EmitNothing_TheThirdRaisesConnectionLost_AndTheNextSuccessCarriesEverythingSince() { var stored = _provider.Pens; _provider.FailPens = true; StartAndEnterTheWait(); Advance(ArchiveReadLoop.ReadInterval * 2); - _messagePanel.Entries.Should().BeEmpty("one reconnect after a server restart must not open the panel"); + _states.Should().BeEmpty("one reconnect after a server restart must not raise the fault"); Advance(ArchiveReadLoop.ReadInterval); - _messagePanel.Entries.Should().ContainSingle(); + _states.Should().ContainSingle(); + ShouldBeConnectionLost(_states[0]); + _messagePanel.Entries.Should().BeEmpty("the bar writes the one entry a lost connection makes"); _provider.FailPens = false; _provider.Pens = [Renamed(stored[0]), stored[1] with { Color = "#0000ff" }]; Advance(ArchiveReadLoop.ReadInterval); + _states.Should().HaveCount(2); + _states[1].Should().Be(ArchiveConnectionState.Connected); _deltas.Should().ContainSingle() .Which.Revised.Select(pen => pen.PenId).Should().Equal(1, 2); } + [Fact] + public void AFourthUnreachableRead_EmitsNothing() + { + StartAndReadNow(); + _provider.FailPens = true; + + Advance(ArchiveReadLoop.ReadInterval * 4); + + _provider.PensQueryCount.Should().Be(5); + _states.Should().HaveCount(2); + _states[0].Should().Be(ArchiveConnectionState.Connected); + ShouldBeConnectionLost(_states[1]); + _messagePanel.Entries.Should().BeEmpty(); + } + + [Fact] + public void TheFirstSuccessAfterTheFault_EmitsConnected_AndCarriesEverythingSince() + { + var stored = _provider.Pens; + StartAndReadNow(); + _provider.FailPens = true; + Advance(ArchiveReadLoop.ReadInterval * 3); + + _states.Should().HaveCount(2); + _states[0].Should().Be(ArchiveConnectionState.Connected); + ShouldBeConnectionLost(_states[1]); + + _provider.FailPens = false; + _provider.Pens = [Renamed(stored[0]), stored[1] with { Color = "#0000ff" }]; + Advance(ArchiveReadLoop.ReadInterval); + + _states.Should().HaveCount(3); + _states[2].Should().Be(ArchiveConnectionState.Connected); + _deltas.Should().ContainSingle() + .Which.Revised.Select(pen => pen.PenId).Should().Equal(1, 2); + _messagePanel.Entries.Should().BeEmpty(); + + Advance(ArchiveReadLoop.ReadInterval); + _states.Should().HaveCount(3, "only the first success after the fault is news"); + } + + [Fact] + public void ASecondOutageAfterARecovery_RaisesConnectionLostAgain() + { + StartAndReadNow(); + _provider.FailPens = true; + Advance(ArchiveReadLoop.ReadInterval * 3); + _provider.FailPens = false; + Advance(ArchiveReadLoop.ReadInterval); + + _provider.FailPens = true; + Advance(ArchiveReadLoop.ReadInterval * 2); + _states.Should().HaveCount(3); + + Advance(ArchiveReadLoop.ReadInterval); + _states.Should().HaveCount(4); + ShouldBeConnectionLost(_states[3]); + _messagePanel.Entries.Should().BeEmpty(); + } + + [Fact] + public void ATableMissingReadAfterTheFault_LeavesTheFault_AndReachesThePanelAtOnce() + { + StartAndReadNow(); + _provider.FailPens = true; + Advance(ArchiveReadLoop.ReadInterval * 3); + + _provider.PensFailureKind = ArchiveFault.TableMissing; + Advance(ArchiveReadLoop.ReadInterval); + + _states.Should().HaveCount(2, "only a success ends the fault"); + _messagePanel.Entries.Should().ContainSingle("the failed reads before it count toward the third in a row") + .Which.View.Should().Be(ArchiveFailureMapper.Map(_tableMissing)); + + _provider.FailPens = false; + Advance(ArchiveReadLoop.ReadInterval); + _states.Should().HaveCount(3); + _states[2].Should().Be(ArchiveConnectionState.Connected); + } + [Fact] public void ASuccess_ResetsTheFailureCount() { @@ -170,7 +271,80 @@ public void ASuccess_ResetsTheFailureCount() _provider.FailPens = true; Advance(ArchiveReadLoop.ReadInterval * 2); - _messagePanel.Entries.Should().BeEmpty("no third failure in a row has happened"); + _states.Should().Equal([ArchiveConnectionState.Connected], "no third failure in a row has happened"); + _messagePanel.Entries.Should().BeEmpty(); + } + + [Fact] + public void ThreeTableMissingReads_EmitNoState_AndAddOnePanelEntry() + { + StartAndReadNow(); + _provider.FailPens = true; + _provider.PensFailureKind = ArchiveFault.TableMissing; + + Advance(ArchiveReadLoop.ReadInterval * 2); + _messagePanel.Entries.Should().BeEmpty(); + + Advance(ArchiveReadLoop.ReadInterval); + _messagePanel.Entries.Should().ContainSingle(); + + Advance(ArchiveReadLoop.ReadInterval); + + _messagePanel.Entries.Should().ContainSingle("a repeat coalesces into the newest entry") + .Which.View.Should().Be(ArchiveFailureMapper.Map(_tableMissing)); + _states.Should().Equal(ArchiveConnectionState.Connected); + } + + [Fact] + public void TwoUnreachableReadsThenTableMissing_AddOneTableMissingEntry_AndRestartTheConnectionRun() + { + StartAndReadNow(); + _provider.FailPens = true; + Advance(ArchiveReadLoop.ReadInterval * 2); + + _provider.PensFailureKind = ArchiveFault.TableMissing; + Advance(ArchiveReadLoop.ReadInterval); + + _messagePanel.Entries.Should().ContainSingle() + .Which.View.Should().Be(ArchiveFailureMapper.Map(_tableMissing)); + _states.Should().Equal(ArchiveConnectionState.Connected); + + _provider.PensFailureKind = ArchiveFault.Unreachable; + Advance(ArchiveReadLoop.ReadInterval * 2); + _states.Should().Equal([ArchiveConnectionState.Connected], "the TableMissing read ended the connection run"); + + Advance(ArchiveReadLoop.ReadInterval); + _states.Should().HaveCount(2); + ShouldBeConnectionLost(_states[1]); + _messagePanel.Entries.Should().ContainSingle(); + } + + [Fact] + public void ThreeTableMissingReadsThenUnreachable_AddOneTableMissingEntry_AndEmitNoState() + { + StartAndReadNow(); + _provider.FailPens = true; + _provider.PensFailureKind = ArchiveFault.TableMissing; + Advance(ArchiveReadLoop.ReadInterval * 3); + + _provider.PensFailureKind = ArchiveFault.Unreachable; + Advance(ArchiveReadLoop.ReadInterval); + + _messagePanel.Entries.Should().ContainSingle() + .Which.View.Should().Be(ArchiveFailureMapper.Map(_tableMissing)); + _states.Should().Equal(ArchiveConnectionState.Connected); + } + + [Fact] + public void ADeltasSubscriberThatThrows_StillGetsConnectedOutFirst() + { + using var throwing = _sync.Deltas.Subscribe(_ => throw new InvalidOperationException("the subscriber threw")); + _provider.Pens = [Renamed(_provider.Pens[0]), _provider.Pens[1]]; + + StartAndReadNow(); + + _messagePanel.Entries.Should().ContainSingle("the subscriber threw inside the apply"); + _states.Should().Equal(ArchiveConnectionState.Connected); } [Fact] @@ -204,6 +378,7 @@ public void AReadThatThrows_AddsOnePanelEntryAtOnce_AndTheNextReadComesOneInterv Advance(ArchiveReadLoop.ReadInterval); _messagePanel.Entries.Should().ContainSingle() .Which.View.Should().Be(ArchiveFailureMapper.Map(new ExceptionalError(thrown))); + _states.Should().BeEmpty(); _provider.PensReadException = null; _provider.FailPens = true; @@ -212,7 +387,9 @@ public void AReadThatThrows_AddsOnePanelEntryAtOnce_AndTheNextReadComesOneInterv Advance(_tick); _provider.PensQueryCount.Should().Be(4); - _messagePanel.Entries.Should().HaveCount(2, "the throw left two failed reads in a row standing"); + _states.Should().ContainSingle("the throw left two failed reads in a row standing"); + ShouldBeConnectionLost(_states[0]); + _messagePanel.Entries.Should().ContainSingle("a lost connection adds no entry of the loop's own"); } // The third failure in a row lands after disposal, so only the stop check keeps it out of the panel. @@ -233,6 +410,17 @@ public void Dispose_DuringAHeldRead_ReportsNothingWhenTheReadLands() _messagePanel.Entries.Should().BeEmpty(); } + [Fact] + public void Dispose_CompletesTheConnectionStates() + { + var isCompleted = false; + using var subscription = _sync.ConnectionStates.Subscribe(_ => { }, () => isCompleted = true); + + _sync.Dispose(); + + isCompleted.Should().BeTrue(); + } + [Fact] public void Dispose_StopsTheReads() { @@ -254,6 +442,28 @@ private void StartAndEnterTheWait() Advance(_tick); } + // The window's start: the read ReadNow asks for runs on the tick the loop starts. + private void StartAndReadNow() + { + _sync.Start(); + _sync.ReadNow(); + Advance(_tick); + } + + private static void ShouldBeConnectionLost(ArchiveConnectionState state) + { + var expected = FakeDataProvider.BenchArchiveError(ArchiveFault.ConnectionLost, "3"); + + state.Fault.Should().BeEquivalentTo(new + { + expected.Kind, + expected.Host, + expected.Port, + expected.Database, + expected.Detail + }); + } + private void Advance(TimeSpan time) { _scheduler.AdvanceBy(time.Ticks); diff --git a/SemiPlot/SemiPlot.Tests.Unit/UI/Bridge/TrendCoordinatorTests.cs b/SemiPlot/SemiPlot.Tests.Unit/UI/Bridge/TrendCoordinatorTests.cs index e198a5c..f21a251 100644 --- a/SemiPlot/SemiPlot.Tests.Unit/UI/Bridge/TrendCoordinatorTests.cs +++ b/SemiPlot/SemiPlot.Tests.Unit/UI/Bridge/TrendCoordinatorTests.cs @@ -4,11 +4,8 @@ using Microsoft.Reactive.Testing; -using SemiPlot.Core.Data; -using SemiPlot.Core.Data.Errors; using SemiPlot.Core.Trends; using SemiPlot.Tests.Unit.UI.Chart; -using SemiPlot.Tests.Unit.UI.PenEditor; using SemiPlot.UI.Bridge; using Xunit; @@ -110,80 +107,6 @@ public void RealtimeBatches_WithAnEmptyCatalog_EmitNothing() batches.Should().BeEmpty(); } - // The UI scheduler is the coordinator's own, not the provider's, so a state pushed from the data side is - // held until the UI scheduler runs it. A view model binding to this stream therefore never has to marshal. - [Fact] - public void ConnectionFaults_ReachTheConsumerOnTheUiScheduler() - { - var uiScheduler = new TestScheduler(); - var (coordinator, _, provider) = CreateCoordinator(uiScheduler: uiScheduler); - using var ownedCoordinator = coordinator; - var states = new List(); - using var subscription = coordinator.ConnectionFaults.Subscribe(states.Add); - - provider.ReportConnectionState(ArchiveConnectionState.Connected); - states.Should().BeEmpty(); - - uiScheduler.AdvanceBy(1); - - states.Should().Equal(ArchiveConnectionState.Connected); - } - - [Fact] - public void ConnectionFaults_CarryTheProvidersFault() - { - var uiScheduler = new TestScheduler(); - var (coordinator, _, provider) = CreateCoordinator(uiScheduler: uiScheduler); - using var ownedCoordinator = coordinator; - var states = new List(); - using var subscription = coordinator.ConnectionFaults.Subscribe(states.Add); - - provider.ReportConnectionState( - new ArchiveConnectionState(FakePenCatalogueEditor.Refusal(ArchiveFault.ConnectionLost, "3"))); - uiScheduler.AdvanceBy(1); - - states.Should().ContainSingle().Which.IsConnected.Should().BeFalse(); - states[0].Fault!.Kind.Should().Be(ArchiveFault.ConnectionLost); - states[0].Fault!.Detail.Should().Be("3"); - } - - [Fact] - public void Dispose_StopsForwardingTheProvidersConnectionState() - { - var uiScheduler = new TestScheduler(); - var (coordinator, _, provider) = CreateCoordinator(uiScheduler: uiScheduler); - var states = new List(); - using var subscription = coordinator.ConnectionFaults.Subscribe(states.Add); - - coordinator.Dispose(); - provider.ReportConnectionState(ArchiveConnectionState.Connected); - uiScheduler.AdvanceBy(1); - - states.Should().BeEmpty(); - } - - // The two streams are independent: a fault must not stand in for a batch, and a chart holding history - // keeps drawing it while the banner is up. - [Fact] - public void AConnectionFault_LeavesTheRealtimeBatchesUntouched() - { - var (coordinator, scheduler, provider) = CreateCoordinator(realtimeInterval: ChartTestBuilder.RealtimeInterval); - using var ownedCoordinator = coordinator; - var batches = new List(); - using var subscription = coordinator.RealtimeBatches.Subscribe(batches.Add); - - scheduler.AdvanceBy(TrendCoordinator.BatchWindow.Ticks); - - provider.ReportConnectionState( - new ArchiveConnectionState(FakePenCatalogueEditor.Refusal(ArchiveFault.ConnectionLost, "3"))); - - batches.Should().HaveCount(1); - - scheduler.AdvanceBy(TrendCoordinator.BatchWindow.Ticks); - - batches.Should().HaveCount(2); - } - // The provider ending the stream reaches every batch observer: with no onError Rx rethrows it out of the // advance below and takes the emitting thread down. [Fact] @@ -210,18 +133,6 @@ public void AFaultingRealtimeStream_ReachesTheFailureChannelInsteadOfTheSchedule failures.Should().ContainSingle().Which.Should().BeSameAs(thrown); } - [Fact] - public void Dispose_CompletesTheConnectionFaults() - { - var (coordinator, _, _) = CreateCoordinator(); - var isCompleted = false; - using var subscription = coordinator.ConnectionFaults.Subscribe(_ => { }, () => isCompleted = true); - - coordinator.Dispose(); - - isCompleted.Should().BeTrue("a consumer of the stream learns that the window is gone"); - } - // A window failing while disposal runs: OnNext on a disposed subject throws, and Rx turns that into the // OnError the catch inside TryBuildRealtimeBatch exists to prevent. [Fact] @@ -296,7 +207,7 @@ public void AProviderFailureAfterASwitch_ReachesRealtimeFailuresOnce() } private static (TrendCoordinator Coordinator, TestScheduler Scheduler, FakeDataProvider Provider) - CreateCoordinator(TimeSpan? realtimeInterval = null, IScheduler? uiScheduler = null) + CreateCoordinator(TimeSpan? realtimeInterval = null) { var scheduler = new TestScheduler(); var provider = new FakeDataProvider(scheduler, realtimeInterval ?? ChartTestBuilder.RealtimeInterval); @@ -304,7 +215,7 @@ private static (TrendCoordinator Coordinator, TestScheduler Scheduler, FakeDataP provider, provider.Pens, scheduler, - uiScheduler ?? ImmediateScheduler.Instance); + ImmediateScheduler.Instance); return (coordinator, scheduler, provider); } diff --git a/SemiPlot/SemiPlot.Tests.Unit/UI/Chart/TrendChartViewModelTests.cs b/SemiPlot/SemiPlot.Tests.Unit/UI/Chart/TrendChartViewModelTests.cs index 7f384b3..a26e98f 100644 --- a/SemiPlot/SemiPlot.Tests.Unit/UI/Chart/TrendChartViewModelTests.cs +++ b/SemiPlot/SemiPlot.Tests.Unit/UI/Chart/TrendChartViewModelTests.cs @@ -14,7 +14,6 @@ using ScottPlot; -using SemiPlot.Core.Data; using SemiPlot.Core.Trends; using SemiPlot.Tests.Unit.UI.Bridge; using SemiPlot.Tests.Unit.UI.Messages; @@ -71,20 +70,6 @@ public void SetPenVisibility_TogglesPenAndPlottableState() state.Line.IsVisible.Should().BeFalse(); } - [AvaloniaFact] - public void DisposingTheChart_LeavesTheConnectionFaultsForwarding() - { - var (viewModel, _, coordinator, provider) = CreateViewModel(); - var states = new List(); - using var subscription = coordinator.ConnectionFaults.Subscribe(states.Add); - - viewModel.Dispose(); - provider.ReportConnectionState(ArchiveConnectionState.Connected); - - states.Should().Equal(ArchiveConnectionState.Connected); - coordinator.Dispose(); - } - [AvaloniaFact] public void SetActivePen_UpdatesActivePenId() { diff --git a/SemiPlot/SemiPlot.Tests.Unit/UI/Di/TrendWindowBuildTests.cs b/SemiPlot/SemiPlot.Tests.Unit/UI/Di/TrendWindowBuildTests.cs index 1605b25..6b38672 100644 --- a/SemiPlot/SemiPlot.Tests.Unit/UI/Di/TrendWindowBuildTests.cs +++ b/SemiPlot/SemiPlot.Tests.Unit/UI/Di/TrendWindowBuildTests.cs @@ -9,10 +9,7 @@ using ReactiveUI.Avalonia; -using SemiPlot.Core.Data; -using SemiPlot.Core.Data.Errors; using SemiPlot.Tests.Unit.UI.Bridge; -using SemiPlot.Tests.Unit.UI.PenEditor; using SemiPlot.Tests.Unit.UI.Startup; using SemiPlot.UI; using SemiPlot.UI.Localization; @@ -28,7 +25,7 @@ namespace SemiPlot.Tests.Unit.UI.Di; /// /// What TrendWindow.Build wires, driven through the real method: the panel the chart, the -/// minimap and the status bar all report into, the connection stream the bar follows, and the catalogue read loop. +/// minimap and the status bar all report into, and the catalogue read the bar's connection state comes from. /// [Collection(ProcessGlobalStateCollection.Name)] [Trait("Component", "UI")] @@ -71,14 +68,13 @@ public async Task TheChartAndTheMinimap_ReportIntoThePanelTheWindowShows() } [AvaloniaFact] - public async Task TheStatusBar_FollowsTheCoordinatorsConnectionState() + public async Task TheWindowReadsTheCatalogueAtOnce_OnTheRealScheduler() { using var stand = NewArchiveStand(); - var scheduler = stand.Scheduler; var dataProvider = stand.Provider; - var container = stand.Data.ServiceProvider; - var probe = await StartupProbe.ReadAsync(container, StartupProbe.DefaultReadBound); + var probe = await StartupProbe.ReadAsync(stand.Data.ServiceProvider, StartupProbe.DefaultReadBound); + var pensQueriesBefore = dataProvider.PensQueryCount; using var window = TrendWindow.Build( probe.Value, @@ -86,17 +82,8 @@ public async Task TheStatusBar_FollowsTheCoordinatorsConnectionState() AvaloniaScheduler.Instance, App.CsvDialectFor(UiLanguage.En), TimeZoneInfo.Utc); - var mainWindowViewModel = window.ViewModel; - - mainWindowViewModel.StatusBar.IsConnected.Should().BeTrue(); - - dataProvider.ReportConnectionState( - new ArchiveConnectionState( - FakePenCatalogueEditor.Refusal(ArchiveFault.ConnectionLost, "3"))); Dispatcher.UIThread.RunJobs(); - mainWindowViewModel.StatusBar.IsConnected.Should().BeFalse("the bar is bound to the coordinator"); - container.GetRequiredService().Entries.Should().NotBeEmpty( - "the fault reaches the one panel the window shows"); + dataProvider.PensQueryCount.Should().Be(pensQueriesBefore + 1); } } diff --git a/SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/AppStatusBarViewModelTests.cs b/SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/AppStatusBarViewModelTests.cs index 53bfcee..d96a64e 100644 --- a/SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/AppStatusBarViewModelTests.cs +++ b/SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/AppStatusBarViewModelTests.cs @@ -98,8 +98,8 @@ public void AFault_SetsTheFaultStateAndItsText() panel.Entries[0].View.Severity.Should().Be(MessageSeverity.Warning); } - // TrendCoordinator forwards this stream with a bare Subscribe, so an escaping throw would end the - // forwarding subscription and the bar would never move again. + // PenCatalogueSync raises this stream from inside its read loop, ahead of the read's apply, so an escaping + // throw would skip that read's apply. [Fact] public void AThrowInTheHandler_IsReportedAndTheStreamSurvives() { @@ -123,7 +123,7 @@ public void AThrowInTheHandler_IsReportedAndTheStreamSurvives() } // The panel edit is the likeliest thrower of the two, and it is also what the catch reports through, so - // a second throw out of it would escape into the coordinator's bare forwarding subscription. + // a second throw out of it would escape into the catalogue read loop and skip that read's apply. [Fact] public void AThrowOutOfThePanel_IsLoggedAndTheStreamSurvives() { diff --git a/SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/MainWindowTestBuilder.cs b/SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/MainWindowTestBuilder.cs index 869dedd..e10d325 100644 --- a/SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/MainWindowTestBuilder.cs +++ b/SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/MainWindowTestBuilder.cs @@ -131,7 +131,7 @@ public static ArchiveStand NewArchiveStand( /// /// A window built through over one virtual clock and one provider; the - /// catalogue read loop has entered its first wait. + /// catalogue read loop has made its first read. /// public static WindowStand NewWindowStand( IReadOnlyList? pens = null, diff --git a/SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/MainWindowViewModelTests.cs b/SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/MainWindowViewModelTests.cs index 7556580..0ceb1d5 100644 --- a/SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/MainWindowViewModelTests.cs +++ b/SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/MainWindowViewModelTests.cs @@ -95,6 +95,7 @@ public async Task ShowPenEditor_WhenTheReadFails_ReportsOnceAndEmitsNothing() public async Task AnEditorWriteThatLands_ReadsTheCatalogueBeforeTheIntervalPasses() { using var stand = NewWindowStand(penCatalogueEditor: new FakePenCatalogueEditor()); + var pensQueriesBefore = stand.Provider.PensQueryCount; var viewModel = stand.ViewModel; using var penEditor = await OpenPenEditorAsync(viewModel); @@ -102,7 +103,7 @@ public async Task AnEditorWriteThatLands_ReadsTheCatalogueBeforeTheIntervalPasse await penEditor.Groups.CreateGroupCommand.Execute(); Dispatcher.UIThread.RunJobs(); - stand.Provider.PensQueryCount.Should().Be(1); + stand.Provider.PensQueryCount.Should().Be(pensQueriesBefore + 1); } [AvaloniaFact] @@ -115,6 +116,7 @@ public async Task APenWriteThatLands_ReadsTheCatalogueBeforeTheIntervalPasses() ReadResult = Result.Ok(new PenCatalogue([pen], [])) }; using var stand = NewWindowStand(penCatalogueEditor: penCatalogueEditor); + var pensQueriesBefore = stand.Provider.PensQueryCount; var viewModel = stand.ViewModel; using var penEditor = await OpenPenEditorAsync(viewModel); penEditor.SelectedRow = penEditor.Rows.Single(); @@ -125,7 +127,7 @@ public async Task APenWriteThatLands_ReadsTheCatalogueBeforeTheIntervalPasses() Dispatcher.UIThread.RunJobs(); penCatalogueEditor.Changes.Should().ContainSingle(); - stand.Provider.PensQueryCount.Should().Be(1); + stand.Provider.PensQueryCount.Should().Be(pensQueriesBefore + 1); } [AvaloniaFact] @@ -136,6 +138,7 @@ public async Task AnEditorWriteTheEditorRefuses_ReadsNothing() CreateGroupResult = Result.Fail(FakePenCatalogueEditor.Refusal(ArchiveFault.NameTaken, "Gas")) }; using var stand = NewWindowStand(penCatalogueEditor: penCatalogueEditor); + var pensQueriesBefore = stand.Provider.PensQueryCount; var viewModel = stand.ViewModel; using var penEditor = await OpenPenEditorAsync(viewModel); @@ -144,7 +147,7 @@ public async Task AnEditorWriteTheEditorRefuses_ReadsNothing() Dispatcher.UIThread.RunJobs(); penCatalogueEditor.Calls.Should().Contain(new FakeEditorCall.CreateGroup("Gas")); - stand.Provider.PensQueryCount.Should().Be(0); + stand.Provider.PensQueryCount.Should().Be(pensQueriesBefore); } [AvaloniaFact] diff --git a/SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/TrendWindowTests.cs b/SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/TrendWindowTests.cs index cc202e7..8d55b68 100644 --- a/SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/TrendWindowTests.cs +++ b/SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/TrendWindowTests.cs @@ -7,13 +7,11 @@ using SemiPlot.Core.Data; using SemiPlot.Core.Data.Errors; using SemiPlot.Core.Trends; +using SemiPlot.Tests.Unit.UI.Bridge; using SemiPlot.Tests.Unit.UI.Markers; -using SemiPlot.Tests.Unit.UI.PenEditor; -using SemiPlot.Tests.Unit.UI.Startup; -using SemiPlot.UI; using SemiPlot.UI.Bridge; -using SemiPlot.UI.MainWindow; -using SemiPlot.UI.Startup; +using SemiPlot.UI.Localization; +using SemiPlot.UI.Messages; using Xunit; @@ -30,23 +28,15 @@ namespace SemiPlot.Tests.Unit.UI.MainWindow; public sealed class TrendWindowTests { [AvaloniaFact] - public void DisposingTheWindowClosesTheLiveEdgeBeforeTheConnectionStream() + public void DisposingTheWindowClosesTheLiveEdge() { var stand = NewWindowStand(); var provider = stand.Provider; - var connectionObserversAtLiveEdgeClose = new List(); provider.OpenLiveSubscriptionCount.Should().Be(1); - provider.ConnectionFaultsObserverCount.Should().Be(1); - provider.LiveSubscriptionChanging = () => - connectionObserversAtLiveEdgeClose.Add(provider.ConnectionFaultsObserverCount); stand.Dispose(); - connectionObserversAtLiveEdgeClose.Should().Equal( - [1], - "the live edge closed once, with the coordinator's connection subscription still open"); provider.OpenLiveSubscriptionCount.Should().Be(0); - provider.ConnectionFaultsObserverCount.Should().Be(0, "the coordinator is disposed after the chart"); } [AvaloniaFact] @@ -89,25 +79,43 @@ public async Task TheChart_ReadsAndWritesThroughTheMarkerSyncTheWindowStarts() stand.MarkerStore.Writes.Should().Equal(new FakeMarkerWrite.Delete(marker)); } - // The TestScheduler defers a fault reported as the live edge opens past Build, so the bar is bound by then. [AvaloniaFact] - public void AFaultReportedAsTheLiveEdgeOpens_ReachesTheStatusBar() + public void TheWindowReadsTheCatalogueAtOnce() { - using var archive = NewArchiveStand(); - var lost = new ArchiveConnectionState( - FakePenCatalogueEditor.Refusal(ArchiveFault.ConnectionLost, "3")); - archive.Provider.LiveSubscriptionChanging = () => archive.Provider.ReportConnectionState(lost); - using var window = TrendWindow.Build( - archive.Data, - TestLaunch.LauncherAt(AppContext.BaseDirectory), - archive.Scheduler, - App.CsvDialectFor(UiLanguage.En), - TimeZoneInfo.Utc); - archive.Provider.LiveSubscriptionChanging = null; - - archive.Scheduler.AdvanceBy(1); - - window.ViewModel.StatusBar.IsConnected.Should().BeFalse(); + using var stand = NewWindowStand(); + + stand.Provider.PensQueryCount.Should().Be(1, "the connection state starts at window open"); + } + + [AvaloniaFact] + public void ThreeUnreachableCatalogueReads_TurnTheBarToFault_WithOneConnectionLostEntry() + { + using var panel = new MessagePanelViewModel(); + using var stand = NewWindowStand(panel: panel); + var statusBar = stand.ViewModel.StatusBar; + var connectionLost = ArchiveFailureMapper.Map( + FakeDataProvider.BenchArchiveError(ArchiveFault.ConnectionLost, "3")); + stand.Provider.FailPens = true; + + stand.Scheduler.AdvanceBy(ArchiveReadLoop.ReadInterval.Ticks * 3); + + statusBar.IsConnected.Should().BeFalse(); + var entry = panel.Entries.Should().ContainSingle().Which; + entry.View.Should().Be(connectionLost); + + stand.Scheduler.AdvanceBy(ArchiveReadLoop.ReadInterval.Ticks); + + panel.Entries.Should().ContainSingle().Which.Should().BeSameAs(entry); + entry.RepeatCount.Should().Be(1, "a fourth failure neither adds nor restamps an entry"); + + stand.Provider.FailPens = false; + stand.Scheduler.AdvanceBy(ArchiveReadLoop.ReadInterval.Ticks); + + statusBar.IsConnected.Should().BeTrue(); + panel.Entries.Select(message => message.View.Severity).Should().Equal( + MessageSeverity.Info, MessageSeverity.Warning); + panel.Entries[0].View.Title.Should().Be(Resources.StatusConnectionRestoredTitle); + panel.Entries[0].View.Detail.Should().Be(Resources.StatusConnectionRestoredDetail); } [AvaloniaFact] diff --git a/SemiPlot/SemiPlot.Tests.Unit/UI/Markers/MarkerSyncTests.cs b/SemiPlot/SemiPlot.Tests.Unit/UI/Markers/MarkerSyncTests.cs index 42f43a8..f0f0aa5 100644 --- a/SemiPlot/SemiPlot.Tests.Unit/UI/Markers/MarkerSyncTests.cs +++ b/SemiPlot/SemiPlot.Tests.Unit/UI/Markers/MarkerSyncTests.cs @@ -91,7 +91,7 @@ public async Task AReadHeldAcrossASuccessfulWrite_IsDiscarded_AndTheReadNowAfter [Fact] public async Task ADiscardedRead_CountsAsNoFailure() { - _store.ReadResult = Result.Fail>(FakeDataProvider.UnreachableArchive()); + _store.ReadResult = MarkersTableMissing(); StartAndEnterTheWait(); Advance(_readInterval * 2); var heldRead = _store.HoldNextRead(); @@ -177,7 +177,7 @@ public async Task EveryWrite_EndsWithARead_WhateverItsOutcome() [Fact] public void TwoFailedReads_LogTheMarkerRead_TheThirdReportsOnce() { - _store.ReadResult = Result.Fail>(FakeDataProvider.UnreachableArchive()); + _store.ReadResult = MarkersTableMissing(); StartAndEnterTheWait(); Advance(_readInterval * 2); @@ -194,6 +194,20 @@ public void TwoFailedReads_LogTheMarkerRead_TheThirdReportsOnce() _messagePanel.Entries.Should().ContainSingle(); } + [Fact] + public void ThreeUnreachableMarkerReads_AddNoEntry_AndLogEach() + { + _store.ReadResult = Result.Fail>(FakeDataProvider.UnreachableArchive()); + StartAndEnterTheWait(); + + Advance(_readInterval * 3); + + _store.ReadCount.Should().Be(3); + _messagePanel.Entries.Should().BeEmpty("the catalogue read's state carries a lost connection to the bar"); + _logger.Collector.GetSnapshot().Should().HaveCount(3) + .And.AllSatisfy(record => record.Level.Should().Be(LogLevel.Warning)); + } + [Fact] public void DisposingDuringAHeldRead_CancelsTheReadAndReportsNothing() { @@ -225,4 +239,10 @@ private static Result> Stored(params StoredMarker[] { return Result.Ok>(markers); } + + private static Result> MarkersTableMissing() + { + return Result.Fail>( + FakeDataProvider.BenchArchiveError(ArchiveFault.TableMissing, "semiplot_markers")); + } } diff --git a/SemiPlot/SemiPlot.Tests.Unit/UI/Startup/AppMainWindowTests.cs b/SemiPlot/SemiPlot.Tests.Unit/UI/Startup/AppMainWindowTests.cs index 49e6e5e..39d8666 100644 --- a/SemiPlot/SemiPlot.Tests.Unit/UI/Startup/AppMainWindowTests.cs +++ b/SemiPlot/SemiPlot.Tests.Unit/UI/Startup/AppMainWindowTests.cs @@ -40,7 +40,6 @@ public void AStartShowsTheMainWindowOverTheContainersPanelAndClosingItDisposesTh App.ResolveMessagePanel().Should().BeSameAs( viewModel.MessagePanel, "the ReactiveUI handler reaches the panel the window shows"); archive.Provider.OpenLiveSubscriptionCount.Should().Be(1); - archive.Provider.ConnectionFaultsObserverCount.Should().Be(1); window.Show(); Dispatcher.UIThread.RunJobs(); @@ -48,6 +47,5 @@ public void AStartShowsTheMainWindowOverTheContainersPanelAndClosingItDisposesTh Dispatcher.UIThread.RunJobs(); archive.Provider.OpenLiveSubscriptionCount.Should().Be(0, "closing the window disposes the chart"); - archive.Provider.ConnectionFaultsObserverCount.Should().Be(0, "closing the window disposes the coordinator"); } } diff --git a/SemiPlot/SemiPlot.UI/Bridge/ArchiveReadLoop.cs b/SemiPlot/SemiPlot.UI/Bridge/ArchiveReadLoop.cs index 24268d4..81da0de 100644 --- a/SemiPlot/SemiPlot.UI/Bridge/ArchiveReadLoop.cs +++ b/SemiPlot/SemiPlot.UI/Bridge/ArchiveReadLoop.cs @@ -1,3 +1,4 @@ +using System.Globalization; using System.Reactive.Concurrency; using FluentResults; @@ -5,6 +6,7 @@ using Microsoft.Extensions.Logging; using SemiPlot.Core.Data; +using SemiPlot.Core.Data.Errors; using SemiPlot.UI.Messages; namespace SemiPlot.UI.Bridge; @@ -17,9 +19,9 @@ internal static class ArchiveReadLoop } /// -/// Re-reads one archive source every or at once on -/// , hands each successful read to the apply, and reports a failure from the third in a row. -/// Runs on the UI thread, unlocked. +/// Re-reads one archive source every or on , applies +/// each success and reports any other failure than a lost connection from the third in a row. A lost connection only +/// logs; the catalogue loop's state sink carries it to the bar. Runs on the UI thread, unlocked. /// internal sealed class ArchiveReadLoop( Func>> read, @@ -27,7 +29,8 @@ internal sealed class ArchiveReadLoop( string readName, MessagePanelViewModel messagePanel, IScheduler uiScheduler, - ILogger logger) : IDisposable + ILogger logger, + Action? reportConnectionState) : IDisposable { private readonly Func>> _read = read; private readonly Action _apply = apply; @@ -35,10 +38,13 @@ internal sealed class ArchiveReadLoop( private readonly MessagePanelViewModel _messagePanel = messagePanel; private readonly IScheduler _uiScheduler = uiScheduler; private readonly ILogger _logger = logger; + private readonly Action? _reportConnectionState = reportConnectionState; private IDisposable? _loop; private TaskCompletionSource _wake = new(); private int _consecutiveFailures; + private int _consecutiveConnectionFailures; + private ArchiveConnectionState? _lastReportedState; /// Starts the loop; its first read comes one interval later. public void Start() @@ -114,13 +120,41 @@ private async Task ReadAsync(CancellationToken stopped) } _consecutiveFailures = 0; + _consecutiveConnectionFailures = 0; + ReportConnected(); _apply(read.Value); } + private void ReportConnected() + { + if (_reportConnectionState is null || _lastReportedState is { IsConnected: true }) + { + return; + } + + _lastReportedState = ArchiveConnectionState.Connected; + _reportConnectionState(_lastReportedState); + } + private void ReportFailedRead(Result read) { _consecutiveFailures++; + if (read.Errors[0] is ArchiveError { Kind: ArchiveFault.Unreachable } connectionFailure) + { + _consecutiveConnectionFailures++; + _logger.LogWarning( + "The {Read} read failed for lack of a connection, {ConsecutiveConnectionFailures} in a row: {Reason}", + _readName, + _consecutiveConnectionFailures, + connectionFailure.Message); + RaiseFaultOnceReached(connectionFailure); + + return; + } + + _consecutiveConnectionFailures = 0; + if (_consecutiveFailures < ArchiveConnectionState.ConsecutiveFailuresBeforeFault) { _logger.LogWarning( @@ -134,4 +168,22 @@ private void ReportFailedRead(Result read) _messagePanel.ReportFailure(read, _logger); } + + private void RaiseFaultOnceReached(ArchiveError connectionFailure) + { + var isRunComplete = _consecutiveConnectionFailures >= ArchiveConnectionState.ConsecutiveFailuresBeforeFault; + + if (_reportConnectionState is null || !isRunComplete || _lastReportedState is { IsConnected: false }) + { + return; + } + + _lastReportedState = new ArchiveConnectionState(new ArchiveError( + ArchiveFault.ConnectionLost, + connectionFailure.Host, + connectionFailure.Port, + connectionFailure.Database, + ArchiveConnectionState.ConsecutiveFailuresBeforeFault.ToString(CultureInfo.InvariantCulture))); + _reportConnectionState(_lastReportedState); + } } diff --git a/SemiPlot/SemiPlot.UI/Bridge/PenCatalogueSync.cs b/SemiPlot/SemiPlot.UI/Bridge/PenCatalogueSync.cs index db25d00..7f545d5 100644 --- a/SemiPlot/SemiPlot.UI/Bridge/PenCatalogueSync.cs +++ b/SemiPlot/SemiPlot.UI/Bridge/PenCatalogueSync.cs @@ -12,11 +12,12 @@ namespace SemiPlot.UI.Bridge; /// /// Re-reads the pen catalogue and emits what changed since the last successful read, the first time against -/// the catalogue the start sequence read. Everything runs on the UI thread, unlocked. +/// the catalogue the start sequence read, and the archive connection state its reads show. Runs on the UI thread. /// public sealed class PenCatalogueSync : IDisposable { private readonly Subject _deltas = new(); + private readonly Subject _connectionStates = new(); private readonly ArchiveReadLoop> _readLoop; private IReadOnlyList _snapshot; @@ -30,12 +31,25 @@ public PenCatalogueSync( { _snapshot = pens; _readLoop = new ArchiveReadLoop>( - _ => dataProvider.QueryPensAsync(), Apply, "pen catalogue", messagePanel, uiScheduler, logger); + _ => dataProvider.QueryPensAsync(), + Apply, + "pen catalogue", + messagePanel, + uiScheduler, + logger, + _connectionStates.OnNext); } /// Every read that changed something, on the UI thread, in the order read. public IObservable Deltas => _deltas.AsObservable(); + /// + /// Connected at the first success and at the first success after a fault; the fault once per outage, at the + /// th lost connection in a row. On the UI + /// thread. + /// + public IObservable ConnectionStates => _connectionStates.AsObservable(); + /// Starts the loop; its first read comes one interval later. public void Start() { @@ -58,6 +72,7 @@ public void Dispose() { _readLoop.Dispose(); _deltas.OnCompleted(); + _connectionStates.OnCompleted(); } private void Apply(IReadOnlyList read) diff --git a/SemiPlot/SemiPlot.UI/Bridge/TrendCoordinator.cs b/SemiPlot/SemiPlot.UI/Bridge/TrendCoordinator.cs index add6edc..02a2928 100644 --- a/SemiPlot/SemiPlot.UI/Bridge/TrendCoordinator.cs +++ b/SemiPlot/SemiPlot.UI/Bridge/TrendCoordinator.cs @@ -18,15 +18,10 @@ public sealed class TrendCoordinator : IDisposable private readonly IScheduler _uiScheduler; private readonly BehaviorSubject> _penIds; - // Own subject so disposal stops forwarding to every consumer. - private readonly Subject _connectionFaults = new(); - // Written from the data scheduler (a failed window, a provider that ends the stream) and completed from // the UI thread at disposal, so the subject serializes its callers. private readonly ISubject _realtimeFailures = Subject.Synchronize(new Subject()); - private readonly IDisposable _connectionSubscription; - // Every pen set, pens here and each one SetPens pushes, must be dataProvider's own catalogue: the // coordinator subscribes to these identifiers without asking the provider whether it knows them, and a // provider silently drops the ones it does not. @@ -42,10 +37,6 @@ public TrendCoordinator( _penIds = new BehaviorSubject>([.. pens.Select(pen => pen.PenId)]); RealtimeBatches = BuildRealtimeBatches(); RealtimeFailures = _realtimeFailures.ObserveOn(_uiScheduler); - ConnectionFaults = _connectionFaults.AsObservable(); - _connectionSubscription = dataProvider.ConnectionFaults - .ObserveOn(_uiScheduler) - .Subscribe(_connectionFaults.OnNext); } /// @@ -56,28 +47,19 @@ public TrendCoordinator( /// /// Every realtime failure once: a throw inside the batch projection, which the pipeline skips, and a - /// provider that ends the stream. Republished on the UI scheduler, like . + /// provider that ends the stream. Republished on the UI scheduler, like . /// public IObservable RealtimeFailures { get; } - /// - /// The provider's connection state, republished on the UI scheduler so a view model binds to it directly. - /// It never faults; disposal completes it. - /// - public IObservable ConnectionFaults { get; } - /// The pen set the live edge follows: the constructor's, or the last one handed to SetPens. public IReadOnlyList PenIds => _penIds.Value; public void Dispose() { - _connectionSubscription.Dispose(); - // Never disposed: a buffer flush still running on the data scheduler would throw out of // TryBuildRealtimeBatch, and Rx would turn that into the OnError the catch exists to prevent. _realtimeFailures.OnCompleted(); _penIds.OnCompleted(); - _connectionFaults.OnCompleted(); } /// Moves the live edge onto this pen set; the history query covers rows the switch skips. diff --git a/SemiPlot/SemiPlot.UI/Localization/Resources.resx b/SemiPlot/SemiPlot.UI/Localization/Resources.resx index 2b0b7cc..1e3b1e7 100644 --- a/SemiPlot/SemiPlot.UI/Localization/Resources.resx +++ b/SemiPlot/SemiPlot.UI/Localization/Resources.resx @@ -292,10 +292,10 @@ SemiPlot cannot reach the archive - SemiPlot could not open a connection to {0}. + SemiPlot could not open a connection to {0}, or the connection did not answer in time. - Check that the PostgreSQL server is running. Check that the host and port in the connection file are reachable from this machine. Check the route and the firewall. + Check that the PostgreSQL server is running. Check that the host and port in the connection file are reachable from this machine. Check the route and the firewall. Check that the server has a free connection slot (max_connections). The archive refused access diff --git a/SemiPlot/SemiPlot.UI/Localization/Resources.ru.resx b/SemiPlot/SemiPlot.UI/Localization/Resources.ru.resx index d603842..fdeb2a0 100644 --- a/SemiPlot/SemiPlot.UI/Localization/Resources.ru.resx +++ b/SemiPlot/SemiPlot.UI/Localization/Resources.ru.resx @@ -292,10 +292,10 @@ Нет связи с базой данных - Не удалось подключиться к {0}. + Не удалось подключиться к {0}, или подключение не ответило вовремя. - Проверьте, что сервер PostgreSQL запущен. Проверьте, что хост и порт из файла подключения доступны с этой машины. Проверьте маршрут и межсетевой экран. + Проверьте, что сервер PostgreSQL запущен. Проверьте, что хост и порт из файла подключения доступны с этой машины. Проверьте маршрут и межсетевой экран. Проверьте, что у сервера есть свободные подключения (max_connections). База данных отказала в доступе diff --git a/SemiPlot/SemiPlot.UI/MainWindow/AppStatusBarViewModel.cs b/SemiPlot/SemiPlot.UI/MainWindow/AppStatusBarViewModel.cs index 78aff52..8ea3321 100644 --- a/SemiPlot/SemiPlot.UI/MainWindow/AppStatusBarViewModel.cs +++ b/SemiPlot/SemiPlot.UI/MainWindow/AppStatusBarViewModel.cs @@ -81,8 +81,8 @@ public void Dispose() _navigation.WindowChanged -= OnNavigationWindowChanged; } - // TrendCoordinator forwards the connection stream with a bare Subscribe, so a throw out of this handler - // would end that forwarding for the rest of the session rather than reaching an onError. + // PenCatalogueSync raises the state from inside its read loop, ahead of the read's apply, so a throw out of + // this handler would skip that read's apply. private void ApplyConnectionState(ArchiveConnectionState state) { try @@ -97,8 +97,8 @@ private void ApplyConnectionState(ArchiveConnectionState state) return; } - // ArchiveConnectionState: every subscription's first tick reports Connected, so a recovery - // entry is owed only once a fault has been seen. + // The first catalogue read reports Connected too, so a recovery entry is owed only once a fault + // has been seen. if (_hasSeenFault) { var restored = ConnectionRestored(); diff --git a/SemiPlot/SemiPlot.UI/MainWindow/TrendWindow.cs b/SemiPlot/SemiPlot.UI/MainWindow/TrendWindow.cs index d70e78f..d692a48 100644 --- a/SemiPlot/SemiPlot.UI/MainWindow/TrendWindow.cs +++ b/SemiPlot/SemiPlot.UI/MainWindow/TrendWindow.cs @@ -28,9 +28,9 @@ internal sealed class TrendWindow : IDisposable private readonly TrendCoordinator _coordinator; private readonly MarkerSync _markerSync; private readonly TrendChartViewModel _chart; - private readonly AppStatusBarViewModel _statusBar; private readonly MinimapViewModel _minimap; private readonly PenCatalogueSync _catalogueSync; + private readonly AppStatusBarViewModel _statusBar; private readonly NavigationBarViewModel _navigationBar; private readonly TrendLegendViewModel _legend; private readonly PenCatalogueApplier _catalogueApplier; @@ -39,9 +39,9 @@ private TrendWindow( TrendCoordinator coordinator, MarkerSync markerSync, TrendChartViewModel chart, - AppStatusBarViewModel statusBar, MinimapViewModel minimap, PenCatalogueSync catalogueSync, + AppStatusBarViewModel statusBar, NavigationBarViewModel navigationBar, TrendLegendViewModel legend, PenCatalogueApplier catalogueApplier, @@ -50,9 +50,9 @@ private TrendWindow( _coordinator = coordinator; _markerSync = markerSync; _chart = chart; - _statusBar = statusBar; _minimap = minimap; _catalogueSync = catalogueSync; + _statusBar = statusBar; _navigationBar = navigationBar; _legend = legend; _catalogueApplier = catalogueApplier; @@ -95,12 +95,6 @@ public static TrendWindow Build( chart.Navigation.SeedFromArchiveExtent(startupData.Extent); chart.ApplyCatalogue(startupData.Pens); - var statusBar = new AppStatusBarViewModel( - messagePanel, - coordinator.ConnectionFaults, - chart.Navigation, - loggerFactory.CreateLogger()); - var minimapLogger = loggerFactory.CreateLogger(); var minimap = new MinimapViewModel(coordinator, chart, uiScheduler, messagePanel, minimapLogger); var catalogueSync = new PenCatalogueSync( @@ -109,6 +103,13 @@ public static TrendWindow Build( messagePanel, uiScheduler, loggerFactory.CreateLogger()); + + // Subscribed before catalogueSync.Start(), so no state precedes the bar. + var statusBar = new AppStatusBarViewModel( + messagePanel, + catalogueSync.ConnectionStates, + chart.Navigation, + loggerFactory.CreateLogger()); var navigationBar = new NavigationBarViewModel(chart); var legend = new TrendLegendViewModel(chart); var catalogueApplier = new PenCatalogueApplier(catalogueSync, chart, minimap, legend, uiScheduler); @@ -138,6 +139,7 @@ public static TrendWindow Build( launcher); catalogueSync.Start(); + catalogueSync.ReadNow(); markerSync.Start(); markerSync.ReadNow(); chart.RequestInitialHistory(); @@ -147,9 +149,9 @@ public static TrendWindow Build( coordinator, markerSync, chart, - statusBar, minimap, catalogueSync, + statusBar, navigationBar, legend, catalogueApplier, @@ -162,9 +164,9 @@ public void Dispose() _catalogueApplier.Dispose(); _legend.Dispose(); _navigationBar.Dispose(); + _statusBar.Dispose(); _catalogueSync.Dispose(); _minimap.Dispose(); - _statusBar.Dispose(); _chart.Dispose(); _markerSync.Dispose(); _coordinator.Dispose(); diff --git a/SemiPlot/SemiPlot.UI/Markers/MarkerSync.cs b/SemiPlot/SemiPlot.UI/Markers/MarkerSync.cs index 73ba09c..e780f6e 100644 --- a/SemiPlot/SemiPlot.UI/Markers/MarkerSync.cs +++ b/SemiPlot/SemiPlot.UI/Markers/MarkerSync.cs @@ -14,8 +14,9 @@ namespace SemiPlot.UI.Markers; /// -/// Re-reads the stored markers and runs the chart's marker writes one at a time, reporting every failure to the -/// panel. Everything runs on the UI thread, unlocked. +/// Re-reads the stored markers and runs the chart's marker writes one at a time. A read failing for lack of a +/// connection only logs, any other read failure reports from the third in a row; a failed write reports at once. +/// Runs on the UI thread. /// public sealed class MarkerSync : IMarkerChannel, IDisposable { @@ -38,7 +39,7 @@ public MarkerSync( _messagePanel = messagePanel; _logger = logger; _readLoop = new ArchiveReadLoop( - ReadAsync, Apply, "marker", messagePanel, uiScheduler, logger); + ReadAsync, Apply, "marker", messagePanel, uiScheduler, logger, reportConnectionState: null); _archiveCallQueue = new ArchiveCallQueue(() => _writeGeneration++); } diff --git a/SemiPlot/SemiPlot.UI/Messages/ArchiveFailureMapper.cs b/SemiPlot/SemiPlot.UI/Messages/ArchiveFailureMapper.cs index 1f61578..48c4daa 100644 --- a/SemiPlot/SemiPlot.UI/Messages/ArchiveFailureMapper.cs +++ b/SemiPlot/SemiPlot.UI/Messages/ArchiveFailureMapper.cs @@ -177,8 +177,8 @@ private static ArchiveFailureView MapArchive(ArchiveError error) Resources.FormatFailureArchiveTableMissingRemedy(error.Detail), MessageSeverity.Error), - // A lost live edge never opens the startup failure window: it is reported beside a chart that - // keeps its history, so the words say what is still true as well as what failed. + // A lost connection never opens the startup failure window: the status bar reports it beside a chart + // that keeps its history, so the words say what is still true as well as what failed. ArchiveFault.ConnectionLost => new ArchiveFailureView( Resources.FailureArchiveConnectionLostTitle, Resources.FormatFailureArchiveConnectionLostDetail(archive, error.Detail), diff --git a/docs/architecture/charting.md b/docs/architecture/charting.md index 8527db6..ddfb961 100644 --- a/docs/architecture/charting.md +++ b/docs/architecture/charting.md @@ -588,8 +588,8 @@ models, backed by renderer-agnostic models in `SemiPlot.Core`. Responsibilities: - `MainWindow/AppMenuBar` — the File / Edit / View / Help menu. Each checkable item reads its flag `Mode=OneWay` and writes it only through the command it invokes (`AGENTS.md`, UI). - `MainWindow/AppStatusBar` + `AppStatusBarViewModel` — current connection state and the active - aggregation layer, named from resx. Built by `TrendWindow` over the coordinator's connection stream and - the chart's navigation, and writes the fault and recovery entries (`data-integration.md`). + aggregation layer, named from resx. Built by `TrendWindow` over the catalogue sync's `ConnectionStates` + and the chart's navigation, and writes the fault and recovery entries (`data-integration.md`). - `MainWindow/AboutDialog` + `AboutInfo` — the modal naming the product, the assembly version and the configuration directory this run read. - `Settings/SettingsDialog` + `SettingsViewModel` — the modal that edits the `app/` and `connection/` @@ -798,11 +798,12 @@ view model: that hold a layer-0 row of a catalogue pen. `TrendCoordinator.QueryArchiveDaysAsync` is a pass-through to the provider; the time window panel reads it once per open and the export window once per window for their calendars' blackout days (`trend-interaction.md#the-time-window-panel`). -- **Connection state:** `TrendCoordinator.ConnectionFaults` republishes the provider's own - `IObservable` on the UI scheduler. `MainWindow/AppStatusBarViewModel` binds - it and shows it in the status bar, over a chart that keeps its history; every fault it carries - becomes a message-panel entry (see data-integration.md). `TrendCoordinator.Dispose` completes the - republished stream, while the provider's own stream never terminates (`data-integration.md`). +- **Connection state:** `Bridge/PenCatalogueSync.ConnectionStates` carries the + `ArchiveConnectionState` its catalogue reads show, on the UI thread; neither the provider nor the + coordinator carries one. `MainWindow/AppStatusBarViewModel` binds it and shows it in the status bar, + over a chart that keeps its history; a fault becomes one message-panel entry + (`data-integration.md#the-connection-state-the-catalogue-read-reports`). `PenCatalogueSync.Dispose` + completes the stream. These records are the plottables' input shape after the view model maps them onto `EnvelopeColumn` buffers; there is no serialization step. diff --git a/docs/architecture/data-integration.md b/docs/architecture/data-integration.md index 810d6e7..8a83331 100644 --- a/docs/architecture/data-integration.md +++ b/docs/architecture/data-integration.md @@ -41,9 +41,6 @@ public interface IDataProvider // Cold per call: no samples flow until subscribed; the subscriber disposes the returned IDisposable. IObservable> Subscribe(IReadOnlyList penIds); - // Hot, shared by every subscription, never completes and never faults. - IObservable ConnectionFaults { get; } - Task>> QueryPensAsync(); Task>> QueryHistoryAsync( @@ -71,7 +68,7 @@ public interface IDataProvider | `PenHistoryEnvelope` | parallel `Timestamps` / `Min` / `Max` / `Center`, strictly ascending, `NaN` marks a gap | One per pen per history query. | | `ArchiveExtent` | `FirstUtc`, `LastUtc`, `IsEmpty` | The span of the configured variables, consumed by the minimap (`TM-4`). `ArchiveExtent.Empty` is the no-span form. | | `AggregationLayer` | `Raw`, `Minute`, `Hour` | Maps one-to-one onto the archive's `l` column. | -| `ArchiveConnectionState` | `Fault`, `IsConnected` | `Fault` is null while the archive answers and carries the typed error while it does not. | +| `ArchiveConnectionState` | `Fault`, `IsConnected` | `Fault` is null while the archive answers and carries the typed error while it does not. No member of `IDataProvider` returns it: `PenCatalogueSync` derives it from its catalogue reads (The connection state the catalogue read reports, below). | Contract points every implementation keeps: @@ -995,28 +992,69 @@ catalogue. `Dispose` completes the subject. `TrendPenStateTests.Realtime_BreakOneTickAfterTheValue_IsRejectedAndOneMillisecondAfterIt_AddsANaNColumn` pins the step. -### The connection state the poll reports - -`ConnectionFaults` is hot, shared by every subscription and never terminating. - -- Every subscription's first successful tick reports `Connected`, and so does the first success - after a raised fault; ordinary ticks report nothing. That first report is how a consumer knows a - subscription is armed, so nothing filters the stream with `DistinctUntilChanged`. -- A fault is raised after three consecutive failed ticks, not one: Npgsql opens a fresh physical - connection after a reset, so one failed tick is ordinary. The state carries - `ArchiveFault.ConnectionLost` with the host, port, database and the threshold that raised it. -- The fault is raised once per outage and carries the threshold, not a running count. The poll - reports nothing further until a tick succeeds. -- A self-cancelled read is not a failure: disposal's `OperationCanceledException` ends the loop - ahead of the mapper. - -`MainWindow/AppStatusBarViewModel` renders the state: the indicator reads connected or not, and -every fault is mapped by `Messages/ArchiveFailureMapper.Map` into an entry in the message panel. The -bar has a single writer, the stream `TrendWindow.Build` hands its constructor. Because every -subscription's first tick reports `Connected`, the recovery entry is written only once a fault has -been seen — otherwise every launch would announce a connection it never lost. The handler is -wrapped: `TrendCoordinator` forwards this stream with a bare `Subscribe`, so a throw out of the -handler would end the forwarding for the rest of the session instead of reaching an `onError`. +### The connection state the catalogue read reports + +The provider reports no connection state, and the poll keeps none. `Bridge/PenCatalogueSync` derives +the state from its own catalogue reads and exposes it as `ConnectionStates`, an +`IObservable` on the UI thread that its `Dispose` completes. `Bridge/ArchiveReadLoop` +decides each state and emits it through the sink `PenCatalogueSync` hands it. `MarkerSync` hands no sink, +so the marker loop raises no state. + +- A read fails for lack of a connection when its first error is an `ArchiveError` of kind + `ArchiveFault.Unreachable`: a refused or reset socket, a client bound firing, an `NpgsqlException` + without a SQLSTATE, or a server that refuses or ends the session with SQLSTATE class `08`, + `57P01` (admin shutdown), `57P02` (crash shutdown), `57P03` (starting up) or `53300` (too many connections). This is + the connection class. `57014` is not in it: the server answered and ended the statement, so it stays + `QueryTimedOut`. Any other kind, and an error that is not an `ArchiveError`, is not connection class. +- The first successful read emits `Connected`, and so does the first success after a raised fault; + every other success emits nothing. The state goes out before the read is applied, so an apply that + throws leaves the state right. +- Three connection-class failures in a row raise the fault, not one: Npgsql opens a fresh physical + connection after a reset, so one failed read is ordinary. The state carries + `ArchiveFault.ConnectionLost` with the failing error's host, port and database, and the threshold + `ArchiveConnectionState.ConsecutiveFailuresBeforeFault` (3) as its detail, not a running count. +- The fault is raised once per outage. The loop emits nothing more until a read succeeds. A failure of + another kind ends the run of connection-class failures. +- So after the fault, a failure of another kind leaves the bar on the fault while the server answers. + The indicator turns back only at the next success. The other kind's entry reaches the panel at once, + because the failed reads before it count toward the third in a row. +- Each connection-class failure writes one log warning naming the read and its run of connection-class + failures (`ConsecutiveConnectionFailures`). It never reaches the panel from the loop; the bar's one + entry stands for it. A failure of another kind logs its run of failed reads of any kind + (`ConsecutiveFailures`). +- A read that throws instead of answering adds one panel entry at once and leaves both counts as they + were. A read that lands after `Dispose` emits nothing. + +`TrendWindow.Build` calls `ReadNow()` right after `Start()`, so the first `Connected`, or the run of +failures, starts when the window opens (`overview.md#the-read`). Each read waits for a connection under +the 15 s connect bound (`PostgresConnectionSettings.ConnectTimeoutSeconds`) and runs its statement under +its own 5 s bound (Configuration, below), and the loop waits 5 s after each read before the next. The +fault lands with the third failed read of that sequence; how long that takes depends on where in the +wait the outage begins and on which bound ends each read. + +One rule covers the three readers of the shared `NpgsqlDataSource` that repeat on their own: the +catalogue loop, the marker loop and the live-edge poll. A connection-class failure is logged by the +reader that meets it and reaches the operator through this state and the bar's one `ConnectionLost` +entry. Any other kind reaches the panel through the two loops' own route, from the third failure in a +row, coalescing repeats into the newest entry (`overview.md#a-failed-read`). The poll has no route of +its own: it logs every failed tick with its run (`ConsecutiveFailures`) and emits no sample. A +`TableMissing`, an `AccessDenied` or a role `statement_timeout` on `trends` reaches the panel through +the minimap's band reads, every 2 to 60 s and the first failure of each outage +(`trend-interaction.md#archive-overview-minimap`), and through the chart's history read on the next +gesture. The band reads run only while the minimap has an extent and a drawn pen. So an outage makes one +entry from the catalogue state, and the minimap's first failed band read of the outage adds an +`Unreachable` entry of its own. Three cases reach only the log: a stalled marker statement, a poll tick +that keeps exceeding its bound, and a poll failure on `trends` that no band read or gesture repeats +(No failure stops at the log, below). + +`MainWindow/AppStatusBarViewModel` renders the state: the indicator reads connected or not, and a +fault is mapped by `Messages/ArchiveFailureMapper.Map` into one `Warning` entry in the message panel. +The bar has a single writer, `PenCatalogueSync.ConnectionStates`, which `TrendWindow.Build` hands its +constructor before it starts the sync, so no state precedes the subscription. The first catalogue read +reports `Connected` too, so the recovery entry is written only once a fault has been seen; otherwise +every launch would announce a connection it never lost. The handler is wrapped: the sync raises the +state from inside its read loop, ahead of the read's apply, so a throw out of the handler would skip +that read's apply. ## Error semantics @@ -1024,14 +1062,14 @@ handler would end the forwarding for the rest of the session instead of reaching | --- | --- | --- | | Connection refused or DNS failure at startup | failed `Result` | The startup-failure window opens, titled "No connection to the archive", naming the host and port, with a remedy | | Connection lost mid-session | failed `Result` on the query; realtime tick dropped | Chart keeps the data it has | -| Three consecutive realtime ticks fail | `ArchiveFault.ConnectionLost` on `ConnectionFaults`; the observable keeps running | The status indicator turns to the fault state and one `Warning` entry appears; the first tick that succeeds restores the indicator and adds one `Info` entry | +| Three consecutive catalogue reads fail for lack of a connection | `ArchiveFault.ConnectionLost` on `PenCatalogueSync.ConnectionStates`; the loop keeps reading | The status indicator turns to the fault state and one `Warning` entry appears; the first catalogue read that succeeds restores the indicator and adds one `Info` entry | | A column the read needs is absent (`42703`) | `ArchiveFault.ShapeUnexpected` with the server's detail | "The archive has an unexpected shape" — run `semibase site`, then find what altered the table | | Query timeout (`57014`) | `ArchiveFault.QueryTimedOut` | "The archive ended the statement"; `statement_timeout` is the `semiplot` role's own setting | | The database does not exist (`3D000`) | `ArchiveFault.DatabaseMissing` | "The archive is not provisioned" — run `semibase site` | | Credentials refused or a grant missing (`28P01`, `28000`, `42501`) | `ArchiveFault.AccessDenied` | "The archive refused access" — the user, password or grants | | A relation or function a statement needs does not exist (`42P01`, `42883`) | `ArchiveFault.TableMissing` whose detail names every relation that statement touches, or `semiplot_register_new_pens()` | "The archive is not provisioned" — run `semibase site`, which creates them all | -| The catalogue re-read fails mid-session | failed `Result`; `PenCatalogueSync` keeps its snapshot | Nothing for the first two failures in a row, one log warning each; from the third one panel entry, coalesced with the ones after it. The next success applies everything stored since the last success | -| The marker re-read fails mid-session | failed `Result`; `MarkerSync` emits nothing and the chart keeps the markers it shows | As for the catalogue: two log warnings, then one panel entry from the third failure in a row. Against a database without `semiplot_markers` that entry is a `TableMissing` Error about 10 s after the window opens, and the chart works without markers | +| The catalogue re-read fails mid-session | failed `Result`; `PenCatalogueSync` keeps its snapshot | A connection-class failure logs one warning and goes to the bar's `ConnectionLost` entry, never to an entry of its own. Any other kind: nothing for the first two failures in a row, one log warning each; from the third one panel entry, coalesced with the ones after it. The next success applies everything stored since the last success | +| The marker re-read fails mid-session | failed `Result`; `MarkerSync` emits nothing and the chart keeps the markers it shows | As for the catalogue: a connection-class failure logs one warning and goes to the bar's entry, which the catalogue read raises; any other kind writes two log warnings, then one panel entry from the third failure in a row. Against a database without `semiplot_markers` that entry is a `TableMissing` Error about 10 s after the window opens, and the chart works without markers | | `semiplot_tags` present but empty | empty pen list, success | The chart area's own empty state, naming `Edit` -> `Pens and groups` -> `Refresh pen list`: no key has been registered as a pen yet. Not a message-panel entry: an empty catalogue is a state, not something that happened | | Archive present but no rows in the window | success, empty envelope list | Empty chart, no error | | A check constraint refuses a written value (`23514`) | `ArchiveFault.ValueRejected` | "The archive refused the value"; the field reverts, and the message line says why | @@ -1066,13 +1104,13 @@ carries no file-access kind. | `ArchiveFault` | Raised by | Detail | | --- | --- | --- | -| `Unreachable` | a socket failure, a client bound firing, any `NpgsqlException` without a SQLSTATE | empty | +| `Unreachable` | a socket failure, a client bound firing, any `NpgsqlException` without a SQLSTATE, a session the server refuses or ends: SQLSTATE class `08`, `57P01`, `57P02`, `57P03`, `53300` | empty | | `AccessDenied` | `28P01`, `28000`, `42501` | the username | | `DatabaseMissing` | `3D000` | empty | | `TableMissing` | `42P01`, `42883` | every relation the failing statement touches: `trends` for the history and realtime reads, `semiplot_tags, trends` for the archive extent and the archive days, all three catalogue tables for the pen catalogue and every editor write, `semiplot_markers` for every marker statement; `semiplot_register_new_pens()` for the registration | | `ShapeUnexpected` | `42703` | the server's own message | | `QueryTimedOut` | `57014` | empty | -| `ConnectionLost` | three consecutive failed poll ticks | the number of failures that raised it | +| `ConnectionLost` | three consecutive catalogue reads that fail as `Unreachable` | the number of failures that raised it | | `ValueRejected` | `23514` on an editor or marker write | the pen's name, or the marker's label or local moment | | `NameTaken` | `23505` on an editor write | the group name asked for | | `RowGone` | `23503` on an editor write, or an update, rename or delete that touched no row | the pen or group name | @@ -1107,8 +1145,8 @@ and a `MessageSeverity`, and is the one place a remedy is written. The severity mapper rather than at the call site, so a future error type gets one from the `MapUnknown` arm without anyone remembering: `AccessDenied`, `TableMissing`, `DatabaseMissing` and `ShapeUnexpected` are `Error`, because nothing recovers until someone changes a grant, a schema or a provisioning run; -`Unreachable`, `ConnectionLost`, `QueryTimedOut` and `ReadFailed` are `Warning`, because the poll -loop retries by itself. `ValueRejected`, `NameTaken` and `RowGone` are `Warning` too: the operator's +`Unreachable`, `ConnectionLost`, `QueryTimedOut` and `ReadFailed` are `Warning`, because the reads +retry by themselves. `ValueRejected`, `NameTaken` and `RowGone` are `Warning` too: the operator's own edit was refused, and the next edit is the remedy. `MarkerGone` is `Warning` as well: the chart drops the marker, and the operator adds it again if it is still needed. `FailureSeverityTests.EveryArchiveFault_CarriesTheSeverityTheTableNames` pins that split. `ConnectionLost` never opens the startup-failure window; it is an entry in the message panel @@ -1137,10 +1175,27 @@ only the log. That includes the failed history query in `Chart/TrendChartViewMod try-report-continue shape that `Chart/ChartHistoryRequestDebouncer.Deliver` applies to the history it hands on, so one bad batch does not end the realtime subscription. `Deliver` guards that hand-off only: the result it reports instead is reported by a handler that guards itself. The catalogue and -the marker read loops report from the third failure in a row, the live edge's threshold -(`ArchiveConnectionState.ConsecutiveFailuresBeforeFault`); the first two log a warning -(`overview.md#a-failed-read`). The minimap's band and extent reads each report the first failure of an -outage and log every repeat at Warning (`trend-interaction.md#archive-overview-minimap`). +the marker read loops report any failure but a lost connection from the third in a row, the threshold +that also raises the connection fault (`ArchiveConnectionState.ConsecutiveFailuresBeforeFault`); the +first two log a warning (`overview.md#a-failed-read`). The minimap's band and extent reads each report +the first failure of an outage and log every repeat at Warning +(`trend-interaction.md#archive-overview-minimap`). + +A failure for lack of a connection follows one rule in the three readers that repeat on their own, the +catalogue loop, the marker loop and the live-edge poll: the reader logs it, and it reaches the operator +through the catalogue read's state and the status bar's one `ConnectionLost` entry (The connection state +the catalogue read reports, above). The poll's other failures on `trends` reach the panel through the +minimap's band reads and the chart's history reads over the same relation. Three cases stay in the log, +written at every failure: + +- A marker statement stalls while the catalogue answers. The marker read has no bound of its own and + runs to the connection string's 300 s backstop, so after a stall the marker loop can sit on a dead + pooled connection for up to 300 s after the bar has written the recovery entry. +- A poll tick's incremental read exceeds its 10 s bound at every tick. A failed tick does not advance + `lastSeen`, so the next tick repeats the same read, and the live edge stays frozen while the catalogue + answers and the indicator reads connected. +- A poll tick fails with another kind on `trends` while no band read runs, because the archive has no + extent or the minimap draws no pen, and the operator makes no gesture. `Messages/ResultReporting` treats the log and the panel as two independent sinks: the panel edit runs first and the log line from a `finally`, so whichever of the two refuses the failure, the other still @@ -1186,7 +1241,7 @@ indicator, a `Button` bound to the panel's own `ToggleCommand` — the same comm invokes. `overview.md` holds the panel's place in the window and the reach of the ReactiveUI exception observer under it. -`TrendCoordinator.RealtimeBatches` needs one more guard than the connection stream does. Its +`TrendCoordinator.RealtimeBatches` needs a guard of its own. Its projection runs inside `Select`, and Rx turns a throwing projection into `OnError`, which would end the `Publish().RefCount()` stream for every subscriber. `TryBuildRealtimeBatch` catches instead, skips the window and publishes the exception on `TrendCoordinator.RealtimeFailures`. A terminal @@ -1266,8 +1321,9 @@ zones, and no key overrides the zone `[DEC:machine-time-zone]`. The file states no query bound: `statement_timeout` belongs to the `semiplot` role and SemiBase owns it (`postgres-instance.md`). The connection -string carries `Command Timeout=300` as a client backstop; the live-edge poll uses a 10 s bound of -its own on every tick. Loading returns a `Result`; a malformed file is reported at startup, not at +string carries `Command Timeout=300` as a client backstop. Two reads override it with a bound of their +own: the live-edge poll's tick (10 s, `RealtimePoll.CreateTickCommand`) and the catalogue read (5 s, +`PostgresDataProvider.CreateCatalogueCommand`), whose failures raise the connection fault. Loading returns a `Result`; a malformed file is reported at startup, not at first query. The password is stored in plain text; the mitigation is a role that cannot write the archive. @@ -1327,6 +1383,10 @@ process data. `Program.Main` returns 1 once that window closes. - **The read bound sits above the connect timeout** (`PostgresConnectionSettings.ConnectTimeoutSeconds`, 15 s), so an unreachable host fails inside the connect attempt and reports `Unreachable` rather than a timeout. `StartupProbeTests.DefaultReadBound_StaysAboveTheConnectTimeout` pins the ordering. +- **The catalogue read fails as `Unreachable` past 5 s**, its own statement bound (Configuration, + above), ahead of the 30 s caller bound. A server that answers the catalogue more slowly than 5 s at + start, under a lock a provisioning run holds for one, fails the start as `Unreachable` rather than as + `StartupReadTimedOutError`. The failure window's Restart is the remedy; one statement keeps one bound. - **A throw on the startup path is a failed `Result`.** `StartupProbe.ReadAsync` catches, logs with the stack, disposes the container and returns an `ExceptionalError`, which `ArchiveFailureMapper` maps through its `IExceptionalError` arm. diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 39ed17a..bd65db4 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -126,9 +126,9 @@ archive's time zone (`PostgresConnectionSettings.SourceTimeZone`), and construct 2. the marker sync, over the container's `IMarkerStore` (`#the-live-markers`); 3. the chart, whose constructor opens the live edge, seeded from the start sequence's pens and extent, with the marker sync as its marker channel; -4. the status bar, over the coordinator's connection stream and the chart's navigation; -5. the minimap; -6. the catalogue sync; +4. the minimap; +5. the catalogue sync; +6. the status bar, over the catalogue sync's `ConnectionStates` and the chart's navigation; 7. the navigation bar and the legend; 8. the catalogue applier; 9. `MainWindowViewModel`, over the message panel, the status bar, the chart, the minimap, the navigation @@ -138,13 +138,13 @@ archive's time zone (`PostgresConnectionSettings.SourceTimeZone`), and construct panel, a logger, the UI scheduler, the archive zone and the chart navigation's `FirstSample` and `NewestSample`, so the window view model holds none of the export's own inputs. -The coordinator republishes the provider's connection stream through `ObserveOn` on the UI scheduler. -`AvaloniaScheduler` runs a zero-delay action at once when the caller is already on the dispatcher thread, -so `ObserveOn` alone defers nothing for a state emitted on the UI thread. The bar is bound in time for the -first state because the provider emits off the UI thread; the tests' `TestScheduler` defers it. +The status bar subscribes to `ConnectionStates` in its constructor, and the catalogue sync emits a state +only from a read, which none runs before `Start()`. So `Build` builds the bar before it starts the sync, +and no state precedes the bar's subscription +(`data-integration.md#the-connection-state-the-catalogue-read-reports`). -`Build` then starts the catalogue sync, starts the marker sync and asks it for its first read, requests -the initial history and starts the minimap's extent load. A part +`Build` then starts the catalogue sync and asks it for its first read, starts the marker sync and asks +it for its first read, requests the initial history and starts the minimap's extent load. A part is built once and never replaced, so every part of the window view model is non-null. `TrendWindow.Dispose` disposes the same objects in reverse order, the coordinator last, and `MainWindowViewModel` disposes only its own commands and request subjects. A throw out of `Build` ends @@ -334,8 +334,8 @@ one method: nothing else may resolve a service through a static. `TrendCoordinator` buffers them on the data scheduler into a coalesced `RealtimeBatch` (≤ 10 Hz / 100 ms), crosses to the UI scheduler via `ObserveOn`, and exposes them as `IObservable`; the chart view model subscribes and appends to the per-pen - plottables. The same provider reports its own connection state, which the status bar shows and the - message panel records. + plottables. The provider reports no connection state: the catalogue sync derives it from its reads, + and the status bar shows it and the message panel records it (`#a-failed-read`). - **History:** the chart requests a window → `TrendCoordinator.QueryHistoryAsync` (the single history query, reached through the debouncer by the initial load and every gesture alike) → provider returns one server-reduced `PenHistoryEnvelope` per pen (ascending `Timestamps` + `Min`/`Max`/`Center`) → the view @@ -614,19 +614,25 @@ connection it edits is the one every read runs over (`#the-settings-window`). reads, so the read carries the provider's normalisation and nothing repeats it. Its loop waits for `ReadInterval`, 5 s, or for a `ReadNow()`, whichever comes first, and then reads: -- the first read comes one interval after `Start`, because the start sequence has just read the same - catalogue; +- the first read comes at once: `TrendWindow.Build` calls `ReadNow()` right after `Start()`. The start + sequence has just read the same catalogue, but this read starts the connection state, so its first + `Connected`, or its run of failures, starts when the window opens; +- each read runs under the catalogue command's own 5 s statement bound + (`data-integration.md#configuration`), so a stalled server fails it within seconds rather than at the + connection string's 300 s backstop; - reads never overlap; - `ReadNow()` during the wait starts a read at once; during a read it makes exactly one more read follow it; the wait after any read is a full interval. The loop is `Bridge/ArchiveReadLoop`, which `PenCatalogueSync` delegates to with the read name "pen catalogue" and `MarkerSync` runs as well (`#the-live-markers`). It holds the wait, `ReadNow`, the failure -count and its warning; each sync keeps its read and what it does with a result. +counts and their warning, and the connection state it emits through the sink `PenCatalogueSync` hands it; +each sync keeps its read and what it does with a result. The loop runs on the UI scheduler through `ScheduleAsync`. The read is asynchronous I/O and holds no thread, and the loop, `ReadNow`, the snapshot and `Dispose` all run on the UI thread, so nothing is locked. `TrendWindow.Build` builds the sync from the pens the start sequence read, after the -minimap, hands its `ReadNow` to `MainWindowViewModel`, and starts it once the window view model exists. +minimap, hands its `ConnectionStates` to the status bar and its `ReadNow` to `MainWindowViewModel`, and +starts it and asks for the first read once the window view model exists. `Dispose` cancels the wait, and a read that lands after it emits nothing. The editor asks for a read after every write it lands. `ArchiveCallQueue` invokes its success callback @@ -698,13 +704,21 @@ without touching the chart. #### A failed read -A failed read keeps the snapshot, so the next success carries everything since. The first two failures -in a row write a warning to the log. The third failure in a row and every one after it go to the -message panel, and a success resets the count. Three is the live edge's threshold, one constant for -both (`ArchiveConnectionState.ConsecutiveFailuresBeforeFault`), so one reconnect after a server restart -opens no panel. A read that throws instead of answering is a defect, not an -outage: it adds one panel entry at once and leaves the count as it was. A subscriber that throws adds -one panel entry, and the loop reads on; the read it threw on is the baseline all the same. +A failed read keeps the snapshot, so the next success carries everything since. The loop routes a +failure by its kind (`data-integration.md#the-connection-state-the-catalogue-read-reports`): + +- A failure for lack of a connection, `ArchiveFault.Unreachable`, writes a warning to the log and never + reaches the panel from the loop. Three in a row emit the `ConnectionLost` state once per outage; the + status bar turns to the fault and adds one panel entry. The first success after it emits `Connected`, + and the bar adds the recovery entry. +- Any other failure goes to the message panel when it is the third failed read in a row or a later one, + whatever the kinds before it; the first two write a warning to the log. + +A success resets both counts. Three is one constant for both routes +(`ArchiveConnectionState.ConsecutiveFailuresBeforeFault`), so one reconnect after a server restart +opens no panel entry and raises no fault. A read that throws instead of answering is a defect, not an +outage: it adds one panel entry at once and leaves both counts as they were. A `Deltas` subscriber that +throws adds one panel entry, and the loop reads on; the read it threw on is the baseline all the same. #### Why nothing is pushed and nothing is rebuilt whole @@ -714,8 +728,8 @@ few hundred rows every 5 s costs less than the live edge's statement per second, itself arrives 1 to 2 s after the SCADA writes, so 5 s between stations reads as immediate. The chart applies the delta to the pens it has and is never rebuilt whole. A window's parts are built -once by `TrendWindow.Build` and never replaced. The status bar takes the coordinator's connection -stream and the chart's navigation in its constructor, the navigation controller has no way to take a +once by `TrendWindow.Build` and never replaced. The status bar takes the catalogue sync's +`ConnectionStates` and the chart's navigation in its constructor, the navigation controller has no way to take a window back, and a gesture started on a replaced chart would end on a disposed one. ### The live markers @@ -728,7 +742,7 @@ no other route. Nothing is pushed between windows, for the reasons | Part | Holds | | --- | --- | -| `Markers/MarkerSync` | The read loop and the write queue over `IMarkerStore`. It implements `Chart/IMarkerChannel` and reports every failure to the message panel. `TrendWindow` builds and disposes it. | +| `Markers/MarkerSync` | The read loop and the write queue over `IMarkerStore`. It implements `Chart/IMarkerChannel` and reports every failure but a lost connection to the message panel; a lost connection reaches it through the catalogue read's state (`#a-failed-read`). `TrendWindow` builds and disposes it. | | `Chart/IMarkerChannel` | The chart's port: `Reads`, every read no write overtook, on the UI thread, and the four writes, each returning a `Result`. | | `Chart/ChartMarkers` | `TrendChartViewModel.Markers`, created and disposed by the chart view model. It reconciles each read into a `MarkerSet`, runs the gestures' writes and raises `Changed` once per change. | | `Core/Trends/MarkerSet` | The markers sorted by moment, then id, the hold counts and the hit test `NearestWithin`, a binary search. | @@ -741,12 +755,14 @@ would make the two folders use each other. #### The marker read `MarkerSync` runs `Bridge/ArchiveReadLoop` with the read name "marker": every 5 s or at once on -`ReadNow()`, never two reads at a time, as `#the-read` states for the catalogue. Unlike the catalogue it -reads at once when the window opens, because the start sequence reads no marker: `TrendWindow.Build` +`ReadNow()`, never two reads at a time, as `#the-read` states for the catalogue. Like the catalogue it +reads at once when the window opens, here because the start sequence reads no marker: `TrendWindow.Build` calls `Start()` and then `ReadNow()`. Every read that lands emits on `Reads`, changed or not; `MarkerSet.Apply` is the one comparison, and the chart raises `Changed` only when the set differs. A -failed read emits nothing and the chart keeps the markers it shows. The first two failures in a row log -a warning, and the third and every one after it go to the panel (`#a-failed-read`). Against a database +failed read emits nothing and the chart keeps the markers it shows. A failure for lack of a connection +only logs a warning, and the bar's `ConnectionLost` entry, which the catalogue read raises, stands for +it. Any other failure logs a warning for the first two in a row and goes to the panel from the third +(`#a-failed-read`). Against a database provisioned before SemiBase v0.6.0, which has no `semiplot_markers`, that is a `TableMissing` Error about 10 s after the window opens, and the chart works without markers. diff --git a/docs/architecture/testing-strategy.md b/docs/architecture/testing-strategy.md index 6e48430..1214864 100644 --- a/docs/architecture/testing-strategy.md +++ b/docs/architecture/testing-strategy.md @@ -39,7 +39,7 @@ and built from the commit under test. Its value is diagnosis. | The error types' shape and extent arithmetic | `SemiPlot.Tests.Unit/Errors/DataErrorTests.cs`, `Data/ArchiveExtentTests.cs` | | The provider's statement text and its binder | `SemiPlot.Tests.Unit/Postgres/ArchiveStatementTextTests.cs` | | The bucketed fold: one column per bucket, the gap column, the DST drop | `SemiPlot.Tests.Unit/Postgres/BucketedRowFoldTests.cs`, `Postgres/RealArchiveGapTests.cs` | -| The fresh tail's bound, and a tick's own command timeout | `SemiPlot.Tests.Unit/Postgres/FreshTailBoundTests.cs`, `Postgres/ArchiveStatementTextTests.cs` | +| The fresh tail's bound, and the tick's and the catalogue read's own command timeouts | `SemiPlot.Tests.Unit/Postgres/FreshTailBoundTests.cs`, `Postgres/ArchiveStatementTextTests.cs` | | The vendor's observed row shape | `SemiPlot.Tests.Unit/Fixtures/RealArchiveFixtureTests.cs` over `Fixtures/real-archive-rows.csv` | | The shipped configuration set | `SemiPlot.Tests.Unit/DeliveredConfigurationTests.cs` over `ConfigFiles/**`, linked into the output directory by `SemiPlot.Tests.Unit.csproj` | | The live theme's pipeline, its watcher and `App`'s wiring of both | `SemiPlot.Tests.Unit/UI/Settings/AppSectionWatcherTests.cs`, `UI/Startup/AppConfigurationTests.cs`; the wall-clock exception below | @@ -169,9 +169,10 @@ so no plan shape at `Minute` or `Hour` holds either way. The seed look-back clau layer: `ArchiveStatementTextTests` pins it, and `PostgresHistoryReadTests.AWindowWiderThanTheLookBackFloorSeeksBackAsFarAsItAsks` runs it. -**Against a socket, with no server** — `RealtimePollTests` (the live edge's failure ladder, every tick -connecting to `127.0.0.1:1`), `RealtimeSubscriptionDisposalTests` and `ConvergeReadinessTests` (a loopback -listener). They cross the machine boundary and need no container, so they carry no collection and take no +**Against a socket, with no server** — `RealtimePollTests` (a failed tick and its logged run of failures, +every tick connecting to `127.0.0.1:1`), `CatalogueReadUnreachableTests` (the catalogue read over a refused +socket answers `Unreachable` naming the archive), `RealtimeSubscriptionDisposalTests` and +`ConvergeReadinessTests` (a loopback listener). They cross the machine boundary and need no container, so they carry no collection and take no `PostgresContainerFixture`; a missing container runtime does not fail them. `UnreachableArchive.Settings` gives them the unreachable address and an adjustable poll interval. @@ -394,7 +395,8 @@ CI has two jobs: `unit-windows` runs `SemiPlot.Tests.Unit` on `windows-latest`, viewer ships on, and `linux` builds the solution once on `ubuntu-latest` and runs both projects, the runner's Docker daemon serving the fixture. The Windows runner cannot host a Linux container and runs `SemiPlot.Tests.Integration` nowhere but the Linux leg. That includes `RealtimePollTests`, -`RealtimeSubscriptionDisposalTests` and `ConvergeReadinessTests`: they need no container, but they live in +`CatalogueReadUnreachableTests`, `RealtimeSubscriptionDisposalTests` and `ConvergeReadinessTests`: they +need no container, but they live in Integration, so they run only on the Linux leg. ## Ownership diff --git a/docs/architecture/trend-interaction.md b/docs/architecture/trend-interaction.md index af7fac2..2334aa4 100644 --- a/docs/architecture/trend-interaction.md +++ b/docs/architecture/trend-interaction.md @@ -816,11 +816,12 @@ string, null with no drawn pen, and the view converts it with the legend's other read. `Minimap/OutageReport` holds the outage policy, one instance for the band reads in `MinimapBandFeed` and one for the extent reads in `MinimapViewModel`: it reports the first failed read of an outage and logs every further one at Warning, with the exception when the read threw, and the -first successful read ends the outage and arms the report again. The report starts armed. During an outage the catalogue -sync reports every 5 s with its own detail, and the panel coalesces a repeat of its newest entry only, -so a band entry per read would never coalesce; one entry per outage keeps the panel readable. The -catalogue sync answers its 5 s loop the other way: it logs its first two failures in a row and reports -from the third (`overview.md#a-failed-read`). Each read turns a throw into a failed `Result`, so a +first successful read ends the outage and arms the report again. The report starts armed. The panel coalesces a repeat +of its newest entry only, and during an outage other entries land between two band reads, the status +bar's `ConnectionLost` and the chart's failed history reads among them, so a band entry per read would +not coalesce; one entry per outage keeps the panel readable. The catalogue sync reports no lost +connection itself: it logs each one, and three in a row turn the status bar to the fault with one entry +(`overview.md#a-failed-read`). Each read turns a throw into a failed `Result`, so a throw never ends the pipeline and the next read is still scheduled; a throw while a read is applied, the next read's scheduling included, goes to `TryReportFailure` as well. diff --git a/docs/plans/completed/20261019-archive-state-from-catalogue.md b/docs/plans/completed/20261019-archive-state-from-catalogue.md new file mode 100644 index 0000000..a2d5baf --- /dev/null +++ b/docs/plans/completed/20261019-archive-state-from-catalogue.md @@ -0,0 +1,671 @@ +# Archive state from the catalogue read + +## Overview + +- The status bar's archive indicator and its `ConnectionLost` panel entry come from the pen catalogue + read loop, not from the live-edge poll. `PenCatalogueSync` publishes `ArchiveConnectionState`; + `IDataProvider.ConnectionFaults`, its republish in `TrendCoordinator` and the poll's state machine are + deleted. The poll still feeds the live edge with the same statements and samples. +- Only a connection-class failure raises `ConnectionLost`, after three of them in a row. Connection class + is `ArchiveFault.Unreachable`, which this slice widens to the server-side SQLSTATEs that mean "no + connection": class `08`, `57P01`, `57P02`, `57P03` and `53300`. A fired client bound is already + `Unreachable`. `QueryTimedOut` (`57014`) is not connection class: the server answered and ended the + statement, so it keeps its own entry, as do `TableMissing`, `AccessDenied`, `ShapeUnexpected`, + `DatabaseMissing` and `ReadFailed`. At HEAD the poll turns any failure, a missing `trends` table + included, into `ConnectionLost` (`RealtimePoll.cs:225-250`). +- One rule covers the three readers of the shared `NpgsqlDataSource`: the catalogue loop, the marker loop + and the poll. A connection-class failure reaches the operator through the catalogue's state and the one + `ConnectionLost` entry the bar writes; each reader logs its own. Every other kind reaches the panel + through the two loops' own route. The poll has no route of its own any more: its other kinds surface + through the minimap's band reads and the chart's history reads over the same `trends` relation, which run + only while the minimap has an extent and a drawn pen, or on a gesture. So an outage makes one entry from + the read loops and the bar; the minimap's first failed band read of the outage adds its own + `Unreachable` entry, as at HEAD. +- The catalogue read gets its own 5 s statement bound. At HEAD it inherits the connection string's 300 s + backstop, so a stalled server would raise the fault after about 15 minutes. +- The window issues one catalogue read at start, so the failure run, and the first `Connected`, start at + window open instead of 5 s later. +- This prepares the roadmap's later slices: the poll can leave `IDataProvider` and then be deleted without + taking the archive indicator with it. + +## Context (from discovery) + +- Roadmap: docs/plans/roadmaps/20261019-opc-live-edge-roadmap.md — slice archive-state-from-catalogue +- Verified at `36d5d85`. `master` at `cf21d86` carries the same `SemiPlot/`, `docs/` and `AGENTS.md` + trees (`git diff --stat 36d5d85 cf21d86` prints nothing for them). Line numbers below are at `36d5d85`. +- The state type: `ArchiveConnectionState(ArchiveError? Fault)`, `ConsecutiveFailuresBeforeFault = 3`, + `Connected`, `IsConnected` (`SemiPlot/SemiPlot.Core/Data/ArchiveConnectionState.cs:10-18`). Its doc speaks + of "a provider", "every subscription's first tick" and `DistinctUntilChanged` (`:5-9`). +- The member that goes: `IObservable ConnectionFaults { get; }` + (`SemiPlot/SemiPlot.Core/Data/IDataProvider.cs:12-15`). +- Its producer: `PostgresDataProvider` holds `_connectionFaultsSource` and a synchronized + `_connectionFaults` (`SemiPlot/SemiPlot.DataSource.Postgres/PostgresDataProvider.cs:49-53,80,87`), + `PollAsync` builds a `RealtimePoll` with `_settings` (`:374-380`) and pushes `tick.StateChange` + (`:400-403`), and `Dispose` only completes the subject (`:246-260`). `RealtimePoll` builds the state: + `RealtimeTick(Samples, StateChange)` (`RealtimePoll.cs:20`), `Succeed` (`:210-223`), `Fail` (`:225-250`), + which counts every failure kind and builds `ConnectionLost` from `settings.Host/Port/Database` + (`:244-249`), the only read of the `settings` parameter (`:30`). +- Its consumers: `TrendCoordinator` republishes it through `ObserveOn(_uiScheduler)` + (`SemiPlot/SemiPlot.UI/Bridge/TrendCoordinator.cs:21-22,28,45-48,63-67,74,80`), and the + `RealtimeFailures` summary names it in a `cref` (`:59`); `TrendWindow.Build` hands + `coordinator.ConnectionFaults` to `AppStatusBarViewModel` (`SemiPlot/SemiPlot.UI/MainWindow/TrendWindow.cs:98-102`). +- The bar: `AppStatusBarViewModel.ApplyConnectionState` sets `IsConnected`, reports a fault through + `_messagePanel.ReportFailure`, and writes the `Info` recovery entry only after a fault (`_hasSeenFault`) + (`SemiPlot/SemiPlot.UI/MainWindow/AppStatusBarViewModel.cs:86-120`). `IsConnected` starts `true` (`:60`). + Its comments name `TrendCoordinator`'s bare `Subscribe` (`:84-85`) and "every subscription's first tick" + (`:100-101`). +- The loop: `ArchiveReadLoop(read, apply, readName, messagePanel, uiScheduler, logger)` + (`SemiPlot/SemiPlot.UI/Bridge/ArchiveReadLoop.cs:24-30`). One counter `_consecutiveFailures` (`:41`); + `ReadNow()` completes the wake before the loop runs (`:50-53`); `ReadAsync` resets the count and applies + on success (`:100-118`); `ReportFailedRead` logs the first two failures and sends the third and every + later one to the panel whatever the kind (`:120-136`); a thrown read goes to `TryReportFailure` and leaves + the count (`:81-84`). +- The loop's two users: `PenCatalogueSync` (`SemiPlot/SemiPlot.UI/Bridge/PenCatalogueSync.cs:32-33`) and + `MarkerSync` (`SemiPlot/SemiPlot.UI/Markers/MarkerSync.cs:40-41`). +- The window: the bar is built at `TrendWindow.cs:98-102`, the catalogue sync at `:106-111`; + `catalogueSync.Start()` at `:140`, `markerSync.Start()` and `markerSync.ReadNow()` at `:141-142`. + `Dispose` disposes the sync at `:165` and the bar at `:167`; the class doc promises reverse build order + (`:22-25`); fields and constructor parameters follow the build order (`:28-60`). +- The catalogue read: `QueryPensAsync` opens a pooled connection and runs `ArchiveStatements.PenCatalog` + on a command with no `CommandTimeout` (`PostgresDataProvider.cs:105-126`), so it takes the connection + string's `CommandTimeout = 300` (`SemiPlot/SemiPlot.DataSource.Postgres/Configuration/PostgresConnectionSettings.cs:26-28,42`). + The connect bound is `Timeout = 15` (`:20-23,43`). The statement is one `GROUP BY` over the three catalogue + tables (`ArchiveStatements.cs:160-171`). The start sequence reads it too (`SemiPlot/SemiPlot.UI/Startup/StartupProbe.cs:63`), + under a 30 s caller bound (`:22-24`). +- The poll's bound, the precedent: `TickCommandTimeoutSeconds = 10` and `CreateTickCommand` + (`RealtimePoll.cs:37-39,85-88`), pinned by + `ArchiveStatementTextTests.ATicksStatementCarriesItsOwnBoundAndNotTheDataSourceBackstop` + (`SemiPlot/SemiPlot.Tests.Unit/Postgres/ArchiveStatementTextTests.cs:219-233`). +- The fault classes: `ArchiveExceptionMapper.IsConnectionFailure` maps `NpgsqlException`, `SocketException` + and `TimeoutException` to `Unreachable`; the comment states that a fired client bound is one of them + (`SemiPlot/SemiPlot.DataSource.Postgres/ArchiveExceptionMapper.cs:72-87`). `MapSqlState` sends every + unlisted SQLSTATE to `ReadFailed` (`:89-104`), so `57P01` (admin shutdown), `57P03` (starting up) and + `53300` (too many connections) are `ReadFailed` at HEAD. `MapWrite` falls through to `Map` (`:38-47`). + `ArchiveExceptionMapperTests` pins the socket and the client bound at `Unreachable` + (`SemiPlot/SemiPlot.Tests.Unit/Postgres/ArchiveExceptionMapperTests.cs:29-44`). Every `ArchiveError` + carries `Host`, `Port`, `Database` (`SemiPlot/SemiPlot.Core/Data/Errors/ArchiveError.cs:10-21`). +- The panel coalesces a report only into an identical newest entry (`SemiPlot/SemiPlot.UI/Messages/MessagePanelViewModel.cs:54-79,86-89`). +- `ConnectionLost` maps to a `Warning` titled "The archive stopped answering" + (`SemiPlot/SemiPlot.UI/Messages/ArchiveFailureMapper.cs:180-186`), under a comment about "a lost live + edge" (`:180-181`); the resx texts name no poll (`Resources.resx:324-331`). `ArchiveError.Describe` says + "The live edge of the archive stopped answering" (`ArchiveError.cs:37-38`), and `ArchiveFault.ConnectionLost`'s + doc says "poll ticks" (`ArchiveFault.cs:27`). +- The minimap: `MinimapBandFeed` schedules its next band read before it applies a result, so its reads keep + running through failures while a pen is drawn (`SemiPlot/SemiPlot.UI/Minimap/MinimapBandFeed.cs:186-198`), + and `OutageReport` writes the first failure of each outage and logs the rest (`SemiPlot/SemiPlot.UI/Minimap/OutageReport.cs:10-33`). + The band reads `trends` through `QueryHistoryAsync` (`MinimapBandFeed.cs:161-170`). +- Test drivers that read the provider's stream or count catalogue reads: + - unit: `FakeDataProvider.ConnectionFaults`, `ConnectionFaultsObserverCount`, `ReportConnectionState` + (`SemiPlot/SemiPlot.Tests.Unit/UI/Bridge/FakeDataProvider.cs:33-35,167-185`), used by `TrendCoordinatorTests` + (five cases, `:116,133,151,168,214`), `TrendChartViewModelTests` (`:75-85`), `TrendWindowTests` + (`:32-50,92-111`), `TrendWindowBuildTests` (`:27-31,74-100`) and `AppMainWindowTests` (`:43,51`). + `FakeDataProvider.FailPens` answers `Unreachable` (`:52,282-286`). `MainWindowViewModelTests` asserts + absolute `PensQueryCount` values after `NewWindowStand` (`:105,128,147`), whose `AdvanceBy(1)` + (`UI/MainWindow/MainWindowTestBuilder.cs:149`) runs a read that `ReadNow()` asked for. + - unit tests over the loop that fail with `Unreachable`: `PenCatalogueSyncTests` (`:141-174,195-215`) and + `MarkerSyncTests.ADiscardedRead_CountsAsNoFailure` (`:92-110`) and `TwoFailedReads_LogTheMarkerRead_TheThirdReportsOnce` + (`:178-196`). + - integration: `ArmedGate` counts `Connected` states as armed subscriptions + (`SemiPlot/SemiPlot.Tests.Integration/ArmedGate.cs:5-75`) for `RealtimeSubscriptionTests` (`:43,84`), + `RealtimeEmptyArchiveTests` (`:80`) and `LiveEdgeArchiveJourneyTests` (`:184,211`); + `RealtimeSubscriptionTests.DisposingASubscriptionStopsItsPoll` already passes a `FakeLoggerProvider` as + the one logger provider (`:38-39,118-121`); `RealtimeSubscriptionDisposalTests` calibrates its wait on the + running provider's `ConnectionLost` and asserts the dropped provider raised no state (`:30-47`); + `RealtimePollTests` (`:26-49`), `RealtimePollReadTests` (`:39-47,77,94,144,207,214-278`) and + `RealtimeEmptyArchiveTests` (`:33-41,56-65`) read `RealtimeTick`; `RealtimeArchive.TickAsync` and a local + `ConsecutiveFailuresBeforeFault` serve them (`RealtimeArchive.cs:25-27,53-63`); `PostgresCompositionTests` + asserts that `Dispose` completes the stream (`SemiPlot/SemiPlot.Tests.Unit/Postgres/PostgresCompositionTests.cs:135-160`). +- The poll's log lines that stay: the baseline lines (`RealtimePoll.cs:129-134`), the per-tick count line + with `RowCount` (`:190-197`), the failure warning with `ConsecutiveFailures` (`:231-235`), written inside + `ReadOnceAsync` before `PollAsync` checks for cancellation (`PostgresDataProvider.cs:388-393`). + `ArchiveProviderFactory.Build` takes one optional `ILoggerProvider` and sets `Debug` with it + (`SemiPlot/SemiPlot.Tests.Integration/ArchiveProviderFactory.cs:29-45`); `UnreachableArchive.Services` + takes none and sets no level (`SemiPlot/SemiPlot.Tests.Integration/UnreachableArchive.cs:25-31`). + +## Development Approach + +- **testing approach**: Regular (code first, then tests in the same task). Task 4 runs its new + `TrendWindowTests` cases once before rewiring `TrendWindow.Build`, to see them fail. +- Complete each task fully before moving to the next. Each task leaves `dotnet build SemiPlot.slnx` green. +- Every task includes new or updated tests for the code it changes, success and failure paths both. +- All tests pass before the next task starts. +- Update this plan file when scope changes during implementation. +- `dotnet build SemiPlot.slnx` holds at zero warnings (`TreatWarningsAsErrors`, which also makes CS9113 and + IDE0005 errors). `dotnet format SemiPlot.slnx --verify-no-changes` and `dotnet terse` pass over the + touched `.cs` files. + +## Testing Strategy + +- Unit tests (`SemiPlot.Tests.Unit`, no container): + - `PenCatalogueSyncTests`: the state sequence over a `TestScheduler`; the connection-class run against + other kinds; the state lands before the apply. + - `MarkerSyncTests`: a connection-class failure logs only; another kind still reports from the third. + - `ArchiveExceptionMapperTests`: the widened `Unreachable` SQLSTATEs. + - `ArchiveStatementTextTests`: the catalogue command carries its own 5 s bound. + - `TrendWindowTests`: three failed catalogue reads turn the bar to fault with one panel entry; the first + catalogue read happens at start. + - `AppStatusBarViewModelTests`, `AppStatusBarViewTests`: unchanged inputs (a `Subject`), kept green. +- Integration tests (`SemiPlot.Tests.Integration`): + - a socket class without a collection: `QueryPensAsync` over `UnreachableArchive.Services` answers + `ArchiveError { Kind: Unreachable }`, the real-socket proof of the route. + - `ArmedGate` turns into a forwarding logger provider that counts the poll's baseline lines. + - `RealtimeSubscriptionDisposalTests` calibrates on the poll's third failure warning. + - The poll's state assertions go; its sample assertions stay. +- Every new test class carries the `Component`, `Area` and `Category` traits. AwesomeAssertions throughout. + Errors are asserted by type and structured field, never by message wording. +- No UI e2e suite exists beyond the headless tests above. + +## Acceptance Evidence + +- Defect at HEAD, by reading: + - `ArchiveReadLoop.ReportFailedRead` sends the third `Unreachable` catalogue failure to the panel and emits + no state (`ArchiveReadLoop.cs:120-136`); the poll sends its own `ConnectionLost` beside it + (`RealtimePoll.cs:237-249`). Two entries per outage, from two counters. + - `RealtimePollReadTests.TheFirstSuccessAfterARaisedFaultReportsConnectedExactlyOnce` passes at HEAD and + asserts `ConnectionLost` for a renamed `trends` table, which is `TableMissing` (`RealtimePollReadTests.cs:214-278`). + - The catalogue command carries no `CommandTimeout` (`PostgresDataProvider.cs:110`). + - `57P03` maps to `ReadFailed` (`ArchiveExceptionMapper.cs:103`), so a reconnect during a restart's recovery + would bypass a rule keyed on `Unreachable`. +- Defect by test: the new `TrendWindowTests` case + `ThreeUnreachableCatalogueReads_TurnTheBarToFault_WithOneConnectionLostEntry` runs red against the + unchanged `TrendWindow.Build` (Task 4, first step): `IsConnected` stays `true`, because the bar still + follows the coordinator, and the panel holds no entry, because Task 1 already stopped the loop from + reporting `Unreachable`. +- Guards: + - deleting `TrendCoordinator.ConnectionFaults` (Task 4) fails the build with one error per remaining + consumer, and that build lists them; + - by Task 5 no consumer of `IDataProvider.ConnectionFaults` is left, so its guard is + `git grep -n "ConnectionFaults" -- SemiPlot`, run before the deletion (only the interface and its two + implementations) and after it (nothing); + - no new error type is declared, so `FailureSeverityTests` is not engaged by this slice. +- After the change, these commands pass: + +```powershell +dotnet test SemiPlot/SemiPlot.Tests.Unit/SemiPlot.Tests.Unit.csproj --filter "FullyQualifiedName~PenCatalogueSyncTests|FullyQualifiedName~MarkerSyncTests|FullyQualifiedName~TrendWindowTests|FullyQualifiedName~TrendWindowBuildTests|FullyQualifiedName~MainWindowViewModelTests|FullyQualifiedName~ArchiveStatementTextTests|FullyQualifiedName~ArchiveExceptionMapperTests|FullyQualifiedName~AppStatusBar|FullyQualifiedName~TrendCoordinatorTests" +dotnet test SemiPlot/SemiPlot.Tests.Integration/SemiPlot.Tests.Integration.csproj --filter "FullyQualifiedName~Realtime|FullyQualifiedName~LiveEdgeArchiveJourneyTests|FullyQualifiedName~CatalogueReadUnreachableTests" +``` + +- What the unit run must show: + - `PenCatalogueSyncTests`: over `Start(); ReadNow()` and `FailPens` from the second read, the states are + exactly `[Connected, ConnectionLost(detail "3")]` after the fourth read, and `[.., Connected]` after the + first success that follows. The panel stays empty throughout. + - With `PensFailureKind = TableMissing`, no state follows the first `Connected`, and the panel holds one + `TableMissing` entry from the third failure on. `[TableMissing x3, Unreachable]` emits no state. + - `TrendWindowTests`: after three failed reads, `StatusBar.IsConnected` is `false` and the panel holds one + entry whose view equals `ArchiveFailureMapper.Map` of a `ConnectionLost` error. A fourth failure adds no + entry and does not restamp it. The next success sets `IsConnected` and adds the `Info` recovery entry. + - `ArchiveStatementTextTests`: the catalogue command's `CommandTimeout` is `5`. + - `ArchiveExceptionMapperTests`: `08006`, `57P01`, `57P02`, `57P03` and `53300` map to `Unreachable`; + `57014` still maps to `QueryTimedOut`. +- The integration run must show the poll's batches unchanged: every `RealtimePollReadTests` sample + assertion and both `RealtimeSubscriptionTests` cases pass on the log gate. +- Integration runs need a container runtime for the collection classes. ASSUMPTION: Docker is available on + the executing machine. The CI `linux` leg runs the integration project in any case. + +## Progress Tracking + +- Mark completed items with `[x]` immediately when done. +- Add newly discovered tasks with ➕ prefix. +- Document issues or blockers with ⚠️ prefix. +- Keep the plan in sync with the work done. + +## Solution Overview + +- **The catalogue loop decides the state from the read.** `ArchiveReadLoop` takes an optional + `Action? reportConnectionState`. `PenCatalogueSync` passes its own subject's + `OnNext` and exposes the subject as `ConnectionStates`; `MarkerSync` passes `null`. With a sink, the loop + emits `Connected` on the first success and on the first success after a raised fault, and + `ConnectionLost` once per outage, after three connection-class failures in a row. +- **One rule for every reader of the data source.** A connection-class failure is logged by the reader + that met it and reaches the operator through the catalogue's state and the bar's one entry. Any other + kind reaches the panel through the two loops' own route, from the third failure in a row, coalescing + repeats into the newest entry, as today. The poll has no route of its own: a `TableMissing`, an + `AccessDenied` or a role `statement_timeout` on `trends` surfaces through the minimap's band reads (every + 2 to 60 s, the first failure of each outage, `MinimapBandFeed.cs:30-31,206-211`) and through the chart's + history read on the next gesture. The band reads run only while the minimap has an extent and a drawn pen + (`MinimapBandFeed.cs:141-157`). +- **The residual cases, each logged at every failure:** + - the marker loop: a marker statement stalled to the 300 s backstop while the catalogue answers; the marker + read has no bound of its own (`PostgresMarkerStore.cs:42-43`), so after a stall the marker loop can sit + on a dead pooled connection for up to 300 s after the bar has written the recovery entry; + - the poll: a tick whose incremental read exceeds its 10 s bound and repeats every second, because a failed + tick does not advance `_lastSeen`; HEAD does not recover from it either, it raises a permanent + `ConnectionLost` over a frozen edge; + - the poll: a non-connection failure on `trends` while no band read runs (an archive with no extent, or no + drawn pen) and the operator makes no gesture; HEAD showed it as a misworded `ConnectionLost`. + - The poll is deleted with the `opc-live-source` slice, which gives the live source a failure route of its + own. +- **The mapper decides connection class.** `ArchiveExceptionMapper` maps SQLSTATE class `08` and `57P01`, + `57P02`, `57P03`, `53300` to `Unreachable`. The fired client bound stays `Unreachable` and `57014` stays + `QueryTimedOut`. The change applies to every read and, through `MapWrite`'s fall-through, to a write that + meets a restart. +- **`PenCatalogueSync` is the source; `TrendWindow.Build` wires it.** `TrendWindow.Build` builds the + catalogue sync ahead of the status bar, hands its `ConnectionStates` to the bar, and calls + `catalogueSync.ReadNow()` after `Start()`, as it does for the markers. +- **The poll and the coordinator stop carrying state.** `IDataProvider.ConnectionFaults`, + `TrendCoordinator.ConnectionFaults`, `PostgresDataProvider`'s subject and `Dispose`, and the state half of + `RealtimePoll` go. The poll keeps its statements, its 10 s tick bound, its samples, its failure warning + and its count. +- **The catalogue read has a bound of its own.** `PostgresDataProvider` builds the catalogue command with + `CommandTimeout = 5`, through an internal factory pinned by a unit test, the way `RealtimePoll.CreateTickCommand` + is pinned. + +## Technical Details + +- `ArchiveReadLoop` state, all touched on the UI thread only: + - `_consecutiveFailures` (exists) counts every failed read and drives the panel route; + `_consecutiveConnectionFailures` counts connection-class failures in a row and resets on a success and + on any other kind; `_lastReportedState`, the last state the sink got or null, serves the sink. + - On success: reset both counts; with a sink, when `_lastReportedState` is not `Connected`, record and + emit `Connected`; then call `_apply`. The state goes out + before the apply, so an apply that throws leaves the state right. + - On a failure whose error is `ArchiveError { Kind: Unreachable }`: increment both counts, reset nothing, + log a warning with the run under the key `ConsecutiveConnectionFailures`. With a sink, when the run reaches `ConsecutiveFailuresBeforeFault` and no + fault is raised, record and emit a `ConnectionLost` state built from the failing error's + `Host`, `Port` and `Database`, its detail `ConsecutiveFailuresBeforeFault.ToString(CultureInfo.InvariantCulture)`. + No connection-class failure goes to the panel from either loop. + - On any other failure: increment `_consecutiveFailures`, reset `_consecutiveConnectionFailures`; below the + threshold log the warning as today, at or past it `ReportFailure` as today. A run + `[Unreachable, Unreachable, TableMissing]` writes one `TableMissing` entry and raises nothing; + `[TableMissing x3, Unreachable]` writes one `TableMissing` entry and raises nothing. + - An error that is not an `ArchiveError` is not connection class. + - A thrown read stays as it is: one `TryReportFailure` entry at once, counts unchanged. +- `ArchiveConnectionState` keeps its shape. Its doc says what a reader of the archive reports about the + connection, names no UI class, and drops the `DistinctUntilChanged` warning and the per-subscription + sentence. The classification and the `ConnectionLost` construction stay private to the loop, their one + caller. +- `ArchiveFault.ConnectionLost` doc: "A run of consecutive reads failed for lack of a connection. Detail is + the number of failures that raised it." `ArchiveFault.Unreachable` doc gains "or a server that refuses the + session (SQLSTATE class 08, 57P01-57P03, 53300)". `ArchiveError.Describe`'s arm: "The {archive} stopped + answering after {detail} consecutive failed reads." The `ConnectionLost` resx texts stay. +- The `Unreachable` resx texts widen to the causes the kind now carries, a fired read bound and a server out + of connection slots among them (`Resources.resx:292-298`, `Resources.ru.resx:292-298`): + - `FailureArchiveUnreachableDetail`: "SemiPlot could not open a connection to {0}, or the connection did + not answer in time." / "Не удалось подключиться к {0}, или подключение не ответило вовремя." + - `FailureArchiveUnreachableRemedy` gains a last sentence: "Check that the server has a free connection + slot (max_connections)." / "Проверьте, что у сервера есть свободные подключения (max_connections)." + - The title stays. +- `ArchiveExceptionMapper.MapSqlState`: `PostgresErrorCodes.AdminShutdown`, `CrashShutdown`, + `CannotConnectNow`, `TooManyConnections` and any SQLSTATE starting with `08` map to `Unreachable`. +- `PostgresDataProvider`: + - `private const int CatalogueCommandTimeoutSeconds = 5;` and + `internal static NpgsqlCommand CreateCatalogueCommand(NpgsqlConnection connection)`, used by + `QueryPensAsync`. + - `PostgresConnectionSettings.CommandTimeoutSeconds`' summary names both overrides: the poll's tick and + the catalogue read. + - The class drops `IDisposable`: `Dispose` had no job but completing the subject. +- Time to the fault after an outage starts, as estimates for the Post-Completion measurement: + - a refused socket: each read fails at once, so the third failed read lands 10 to 15 s in, depending on + where in the 5 s wait the outage begins; + - a stalled server on a pooled connection: the first read waits the 5 s bound and then Npgsql's + cancellation (ASSUMPTION: Npgsql sends a cancel request when `CommandTimeout` fires, waits at most its + 2 s `CancellationTimeout`, and then breaks the connection). The next reads open a new physical connection + unless an idle pooled one is free, and against a paused server that open waits the 15 s connect bound. + So the range is 7 + 5 + 7 + 5 + 7 = 31 s with spare idle connections, up to 7 + 5 + 15 + 5 + 15 = 47 s + without; + - dropped packets with no pooled connection: each read waits the 15 s connect bound, so about + 3 x 15 s + 2 x 5 s = 55 s. The poll at HEAD takes about 3 x (15 s + 1 s) = 48 s here. + - The documentation states the mechanism (three reads 5 s apart, each under the 5 s statement bound and the + 15 s connect bound), not a total. +- The 5 s bound also covers the start sequence's catalogue read (`StartupProbe.cs:63`). A server that + answers the catalogue more slowly than 5 s at start, a lock held by a provisioning run for one, now fails + the start as `Unreachable` ahead of the probe's 30 s `StartupReadTimedOut` bound. The failure window's + retry is the remedy; one statement keeps one bound. +- `TrendWindow.Build` order: coordinator, marker sync, chart, minimap, catalogue sync, status bar (over + `catalogueSync.ConnectionStates` and the chart's navigation), navigation bar, legend, applier, + `MainWindowViewModel`. Then `catalogueSync.Start(); catalogueSync.ReadNow(); markerSync.Start(); + markerSync.ReadNow();`. `Dispose` runs the reverse, so the bar is disposed before the sync. Fields and + constructor parameters follow the new order. The bar subscribes in its constructor, before `Start()`, so + no state precedes it. +- `AppStatusBarViewModel` keeps its handler and its `try/catch`. The comment at `:84-85` names the new + reason: the sync raises the state from inside the read loop, ahead of the read's apply, so a throw out of + the handler would skip that read's apply. The comment at `:100-101` names the first catalogue read as the + first `Connected`. +- `RealtimePoll`: `ReadOnceAsync` returns `IReadOnlyList`, and `RealtimeTick` goes. A failure + returns no sample and logs the warning with its running count, as at `:231-235`. `_faultRaised`, + `_reportedConnected`, the state half of `Succeed` and `Fail`, and the `settings` parameter go; the + constructor call in `PostgresDataProvider.PollAsync` and `RealtimeArchive.NewPoll` drop it. The comment on + `TickCommandTimeoutSeconds` names its remaining job: a stalled tick holds the live edge for at most the + bound. +- The baseline log lines carry the structured key `SubscribedPenCount` in place of `PenCount`, so a test + can tell them from the history read's `PenCount`. Both lines keep their wording otherwise. +- Integration arming: `ArmedGate` implements `ILoggerProvider` over an optional inner provider, to which + it forwards every entry; it counts entries whose structured state carries `SubscribedPenCount`, and + `Armed(n)` completes at the nth such line. Over an archive with no row for the subscribed pens, the + "no archived row yet" line repeats every tick, so the gate counts lines, not subscriptions; every caller + arms its subscriptions one at a time over a seeded archive, or awaits only the first line. The gate also + records the highest `ConsecutiveFailures` it saw, and `RealtimeSubscriptionTests` asserts it stayed below + `3`, the same tolerance HEAD's "no fault seen" had. `FailuresReached(n)` completes when an entry carries + `ConsecutiveFailures >= n`; `RealtimeSubscriptionDisposalTests` awaits it on the running provider. `UnreachableArchive.Services` takes an optional + `ILoggerProvider` and sets `Debug` with it, as `ArchiveProviderFactory.Build` does. +- `FakeDataProvider` gains `ArchiveFault PensFailureKind { get; set; } = ArchiveFault.Unreachable`, read + when `FailPens` is set, through `BenchArchiveError`. It loses `ConnectionFaults`, + `ConnectionFaultsObserverCount` and `ReportConnectionState` in Task 5. + +## What Goes Where + +- **Implementation Steps** (`[ ]` checkboxes): code, tests and documentation in this repository. +- **Post-Completion** (no checkboxes): the timed outage on the demo stand and the remaining slices. + +## Implementation Steps + +### Task 1: Make the catalogue read loop the source of the connection state + +**Files:** +- Modify: `SemiPlot/SemiPlot.UI/Bridge/ArchiveReadLoop.cs` +- Modify: `SemiPlot/SemiPlot.UI/Bridge/PenCatalogueSync.cs` +- Modify: `SemiPlot/SemiPlot.UI/Markers/MarkerSync.cs` +- Modify: `SemiPlot/SemiPlot.Core/Data/ArchiveConnectionState.cs` +- Modify: `SemiPlot/SemiPlot.Core/Data/Errors/ArchiveFault.cs` +- Modify: `SemiPlot/SemiPlot.Core/Data/Errors/ArchiveError.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/UI/Bridge/FakeDataProvider.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/UI/Bridge/PenCatalogueSyncTests.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/UI/Markers/MarkerSyncTests.cs` + +- [x] (review 84aafa5 merged the two flags into one `_lastReportedState`) + add the `reportConnectionState` sink, the second counter, `_faultRaised` and `_reportedConnected` to + `ArchiveReadLoop`; emit on success before `_apply`; route a failure by kind as Technical Details + states; update the class summary (`ArchiveReadLoop.cs:19-23`): without a sink, a connection-class + failure only logs, because the catalogue loop's sink carries it to the bar +- [x] give `PenCatalogueSync` a `Subject` exposed as `ConnectionStates`, passed to + the loop and completed in `Dispose`; `MarkerSync` passes `null` (`MarkerSync.cs:40-41`), and its class + summary (`MarkerSync.cs:16-19`) says a connection-class read failure reaches the panel through the + catalogue read's state, every other failure directly +- [x] update the docs of `ArchiveConnectionState`, `ArchiveFault.ConnectionLost` and the `ConnectionLost` + arm of `ArchiveError.Describe` +- [x] add `FakeDataProvider.PensFailureKind` +- [x] rewrite `PenCatalogueSyncTests.TwoFailedReads_ReportNothing_TheThirdReportsOnce_AndTheNextSuccessCarriesEverythingSince` + (`:141-158`) and `AReadThatThrows_AddsOnePanelEntryAtOnce_AndTheNextReadComesOneIntervalLater` + (`:195-215`) for the new route: the third `Unreachable` emits `ConnectionLost` and adds no entry; + `ASuccess_ResetsTheFailureCount` (`:161-174`) asserts that no `ConnectionLost` state is emitted +- [x] write `PenCatalogueSyncTests` cases: the first success emits `Connected` once and later successes emit + nothing; a fourth `Unreachable` emits nothing; the first success after the fault emits `Connected` and + still carries everything since; `TableMissing` three times emits no state and one entry; + `[Unreachable, Unreachable, TableMissing]` and `[TableMissing x3, Unreachable]` each give one + `TableMissing` entry and no state; a deltas subscriber that throws still gets `Connected` out first; + `Dispose` completes `ConnectionStates`; a held read that fails after `Dispose` emits nothing +- [x] `MarkerSyncTests.ADiscardedRead_CountsAsNoFailure` (`:92-110`) and + `TwoFailedReads_LogTheMarkerRead_TheThirdReportsOnce` (`:178-196`) fail with + `BenchArchiveError(ArchiveFault.TableMissing, "semiplot_markers")`; add a case: three `Unreachable` + marker reads add no entry +- [x] run `dotnet test SemiPlot/SemiPlot.Tests.Unit/SemiPlot.Tests.Unit.csproj` - must pass before Task 2 + +### Task 2: Bound the catalogue read and widen the connection class + +**Files:** +- Modify: `SemiPlot/SemiPlot.DataSource.Postgres/PostgresDataProvider.cs` +- Modify: `SemiPlot/SemiPlot.DataSource.Postgres/ArchiveExceptionMapper.cs` +- Modify: `SemiPlot/SemiPlot.DataSource.Postgres/Configuration/PostgresConnectionSettings.cs` +- Modify: `SemiPlot/SemiPlot.Core/Data/Errors/ArchiveFault.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/Postgres/ArchiveStatementTextTests.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/Postgres/ArchiveExceptionMapperTests.cs` +- Modify: `SemiPlot/SemiPlot.UI/Localization/Resources.resx` +- Modify: `SemiPlot/SemiPlot.UI/Localization/Resources.ru.resx` +- Create: `SemiPlot/SemiPlot.Tests.Integration/CatalogueReadUnreachableTests.cs` + +- [x] add `CatalogueCommandTimeoutSeconds = 5` (private) and `CreateCatalogueCommand`; build the command in + `QueryPensAsync` (`:110`) through it +- [x] update the summary of `PostgresConnectionSettings.CommandTimeoutSeconds` (`:26-28`) +- [x] map class `08`, `57P01`, `57P02`, `57P03` and `53300` to `Unreachable` in `MapSqlState` (`:89-104`); + update `ArchiveFault.Unreachable`'s doc (verified 2026-10-19: no test pins `57P0x`, `53300` or a class + `08` SQLSTATE, `git grep -n -E "57P0|53300" -- SemiPlot` prints only unrelated lines) +- [x] widen the `Unreachable` detail and remedy in both resx files as Technical Details states; any + `ArchiveFailureMapperTests` case that compares the rendered text follows +- [x] write `ArchiveStatementTextTests.TheCatalogueCommandCarriesItsOwnBoundAndNotTheDataSourceBackstop`, + restating `5` rather than reading the constant, beside the tick test (`:219-233`) +- [x] write `ArchiveExceptionMapperTests` theory cases for `08006`, `57P01`, `57P02`, `57P03`, `53300` to + `Unreachable`, `57014` to `QueryTimedOut`, and one write through `MapWrite` with `57P01` to `Unreachable` +- [x] write `CatalogueReadUnreachableTests` (no collection, no fixture, like `RealtimePollTests`): + `QueryPensAsync` over `UnreachableArchive.Services` answers `ArchiveError { Kind: Unreachable }` naming + the configured host, port and database +- [x] run the unit project and `dotnet test SemiPlot/SemiPlot.Tests.Integration/SemiPlot.Tests.Integration.csproj --filter "FullyQualifiedName~PostgresCatalogReadTests|FullyQualifiedName~LiveCatalogueTests|FullyQualifiedName~CatalogueReadUnreachableTests"` + - must pass before Task 3 + +### Task 3: Arm the integration tests on the poll's own log lines + +**Files:** +- Modify: `SemiPlot/SemiPlot.DataSource.Postgres/RealtimePoll.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Integration/ArmedGate.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Integration/UnreachableArchive.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Integration/RealtimeSubscriptionTests.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Integration/RealtimeEmptyArchiveTests.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Integration/RealtimeSubscriptionDisposalTests.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Integration/Journeys/LiveEdgeArchiveJourneyTests.cs` + +- [x] rename the baseline lines' key `PenCount` to `SubscribedPenCount` (`RealtimePoll.cs:129-134`) +- [x] rewrite `ArmedGate` as a forwarding `ILoggerProvider` over the baseline lines, recording the highest + `ConsecutiveFailures`; its summary says it counts lines; give `UnreachableArchive.Services` an optional + `ILoggerProvider` that sets `Debug` +- [x] pass the gate to `ArchiveProviderFactory.Build` in `RealtimeSubscriptionTests` (`:39-46,80-97`; in + `DisposingASubscriptionStopsItsPoll` the gate wraps the existing `FakeLoggerProvider`), + `RealtimeEmptyArchiveTests.ASubscriptionOverAnArchiveWithNoRowReportsConnected` (renamed + `...LogsItsBaseline`) and `LiveEdgeArchiveJourneyTests.OpenStandAsync` (`:165,184`); the + `NoFaultWasSeen` assertion (`RealtimeSubscriptionTests.cs:107-109`) reads the highest + `ConsecutiveFailures` below `3`; the class comment (`RealtimeSubscriptionTests.cs:15-17`) names the + baseline log line as the awaited emission in place of `Connected` +- [x] rewrite `RealtimeSubscriptionDisposalTests`: the running provider's gate awaits `FailuresReached(3)`, and after the same margin as HEAD the dropped provider's highest + `ConsecutiveFailures` is below `3` and it delivered no batch +- [x] run the integration filter `FullyQualifiedName~Realtime|FullyQualifiedName~LiveEdgeArchiveJourneyTests` + - must pass before Task 4 + +### Task 4: Feed the status bar from the catalogue sync + +**Files:** +- Modify: `SemiPlot/SemiPlot.UI/MainWindow/TrendWindow.cs` +- Modify: `SemiPlot/SemiPlot.UI/Bridge/TrendCoordinator.cs` +- Modify: `SemiPlot/SemiPlot.UI/MainWindow/AppStatusBarViewModel.cs` +- Modify: `SemiPlot/SemiPlot.UI/Messages/ArchiveFailureMapper.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/TrendWindowTests.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/MainWindowViewModelTests.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/UI/MainWindow/MainWindowTestBuilder.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/UI/Di/TrendWindowBuildTests.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/UI/Startup/AppMainWindowTests.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/UI/Bridge/TrendCoordinatorTests.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/UI/Chart/TrendChartViewModelTests.cs` + +- [x] write `TrendWindowTests.ThreeUnreachableCatalogueReads_TurnTheBarToFault_WithOneConnectionLostEntry` + and `TheWindowReadsTheCatalogueAtOnce`; run them against the unchanged `Build` and record that both fail +- [x] reorder `TrendWindow.Build`, `Dispose`, the fields and the constructor as Technical Details states; + hand `catalogueSync.ConnectionStates` to the bar; add `catalogueSync.ReadNow()` after `Start()` +- [x] delete `TrendCoordinator.ConnectionFaults`, its subject, its subscription and their disposal + (`TrendCoordinator.cs:21-22,28,45-48,63-67,74,80`), and edit the `RealtimeFailures` summary's `cref` + (`:59`) rather than deleting it; build and confirm that the compiler names every remaining consumer, + then fix each +- [x] rewrite the comments in `AppStatusBarViewModel` (`:84-85`, `:100-101`) and `ArchiveFailureMapper` + (`:180-181`) +- [x] ➕ rewrite the two `AppStatusBarViewModelTests` comments (`:101-102`, `:125-126`) that named the + coordinator's bare forwarding subscription +- [x] update the tests that read the coordinator's stream: delete `TrendCoordinatorTests`' five connection + cases (`:116,133,151,168,214`) and `TrendChartViewModelTests.DisposingTheChart_LeavesTheConnectionFaultsForwarding` + (`:75-85`); `TrendWindowTests.DisposingTheWindowClosesTheLiveEdgeBeforeTheConnectionStream` becomes + `DisposingTheWindowClosesTheLiveEdge` over `OpenLiveSubscriptionCount`; `AFaultReportedAsTheLiveEdgeOpens_ReachesTheStatusBar` + (`:92-111`) is replaced by the new fault test; `AppMainWindowTests` drops its two + `ConnectionFaultsObserverCount` assertions (`:43,51`) +- [x] `TrendWindowBuildTests.TheStatusBar_FollowsTheCoordinatorsConnectionState` (`:74-100`) becomes + `TheWindowReadsTheCatalogueAtOnce_OnTheRealScheduler`: it drops the pushed fault and asserts only that + `PensQueryCount` rose by one after `Dispatcher.UIThread.RunJobs()`; the class summary (`:27-31`) names + the catalogue read as the bar's source +- [x] `MainWindowViewModelTests` (`:105,128,147`) asserts the rise of `PensQueryCount` against a count taken + after the stand is built, not an absolute value; check `TrendWindowTests.DisposingTheWindowStopsTheCatalogueRead` + and `TheChart_ReadsAndWritesThroughTheMarkerSyncTheWindowStarts` against the extra read at start; + the comment at `MainWindowTestBuilder.cs:134` says the catalogue loop has made its first read +- [x] remove the usings the deletions leave dead (IDE0005 fails the build): in `TrendCoordinatorTests.cs` + `SemiPlot.Core.Data.Errors` and `SemiPlot.Tests.Unit.UI.PenEditor`; in `TrendWindowBuildTests.cs` + (`:12,13,15`) `SemiPlot.Core.Data`, `.Errors` and `UI.PenEditor`; in `TrendWindowTests.cs` (`:8`) unless + the new fault test uses `ArchiveFault`; whichever the build names +- [x] run the unit project - must pass before Task 5 + +### Task 5: Delete the poll's connection state + +**Files:** +- Modify: `SemiPlot/SemiPlot.Core/Data/IDataProvider.cs` +- Modify: `SemiPlot/SemiPlot.DataSource.Postgres/PostgresDataProvider.cs` +- Modify: `SemiPlot/SemiPlot.DataSource.Postgres/RealtimePoll.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/UI/Bridge/FakeDataProvider.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/Postgres/PostgresCompositionTests.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Integration/RealtimeArchive.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Integration/RealtimePollTests.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Integration/RealtimePollReadTests.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Integration/RealtimeEmptyArchiveTests.cs` + +- [x] run `git grep -n "ConnectionFaults" -- SemiPlot` and record the hits: after Tasks 3 and 4 only the + interface, `PostgresDataProvider` and `FakeDataProvider` remain; delete `IDataProvider.ConnectionFaults` + and its comment (`IDataProvider.cs:12-15`) (recorded: those three plus `PostgresCompositionTests.cs:142`, + which this task deletes; nothing after) +- [x] delete the subject pair, `ConnectionFaults`, `Dispose`, `_disposed` and `IDisposable` from + `PostgresDataProvider` (`:49-53,60,80,83-87,246-260`) and the `StateChange` push from `PollAsync` + (`:400-403`); fix the class summary +- [x] reduce `RealtimePoll` as Technical Details states, the `settings` parameter included, and drop it from + `PostgresDataProvider.PollAsync` (`:374-380`) and `RealtimeArchive.NewPoll` (`RealtimeArchive.cs:42-51`); + rewrite the comment at `RealtimePoll.cs:37-38` +- [x] remove the usings the deletions leave dead (IDE0005 fails the build): `System.Globalization`, the + `SemiPlot.Core.Data`/`.Errors` usings and `SemiPlot.DataSource.Postgres.Configuration` (`:12`) in + `RealtimePoll.cs`; `System.Reactive.Subjects` in `PostgresDataProvider.cs`; `SemiPlot.Core.Data`, + `.Configuration` (`:11`) and `Xunit` (`:14`) in `RealtimeArchive.cs`; `RealtimePollReadTests.cs:5-6`; + `SemiPlot.Core.Data.Errors` in `RealtimePollTests.cs`; `System.Reactive.Disposables` and + `System.Reactive.Subjects` in `FakeDataProvider.cs` (`:3,5`); whichever the build names +- [x] delete `FakeDataProvider.ConnectionFaults`, `ConnectionFaultsObserverCount` and `ReportConnectionState` +- [x] delete `PostgresCompositionTests.DisposingTheProviderCompletesTheConnectionFaultStream` and + `DisposingTwiceDoesNotThrow` (`:135-160`) +- [x] integration: `RealtimeArchive.TickAsync` and its local `ConsecutiveFailuresBeforeFault` go + (`RealtimeArchive.cs:25-27,53-63`); every `.Samples` read becomes the returned list; + `RealtimePollTests` keeps `ASingleFailure...` as `AFailedTickEmitsNoSample` and replaces the ladder case + with one that reads the warning's `ConsecutiveFailures` as 1 to 5 over five ticks; + `RealtimePollReadTests` drops every `StateChange` assertion (`:46,77,94,144,207,251-277`), renames + `TheFirstTickEmitsNoSampleAndReportsTheSubscriptionArmed` to `TheFirstTickEmitsNoSample` and + `TheFirstSuccessAfterARaisedFaultReportsConnectedExactlyOnce` to + `TheFirstTickAfterTheTableReturnsTakesTheBaseline`, keeping its sample assertions and ticking three + times without `TickAsync`; `RealtimeEmptyArchiveTests` drops its `StateChange` assertions (`:36,41`) +- [x] run `dotnet test SemiPlot.slnx` - must pass before Task 6 + +### Task 6: Verify acceptance criteria + +- [x] verify the Overview against the code: `git grep -n "ConnectionFaults" -- SemiPlot` prints nothing; + `git grep -n "ArchiveConnectionState" -- SemiPlot/SemiPlot.DataSource.Postgres` prints nothing + (both print nothing) +- [x] verify the scope guard: `git diff origin/master --name-only` lists only the files of Tasks 1 to 5 and + 7 and this plan; `git diff origin/master -- SemiPlot | grep -i -c "opc"` prints `0`; + `ArchiveStatements.cs` is not in the list (41 files of Tasks 1-5 plus this plan, Task 7 not run yet; + `opc` count 0; no `ArchiveStatements.cs`) +- [x] run both Acceptance Evidence commands and check each "must show" item (unit filter 159/159, + integration filter 20/20; every item matched to an assertion) +- [x] run `dotnet build SemiPlot.slnx` (0 warnings) and `dotnet format SemiPlot.slnx --verify-no-changes` + (`--no-incremental`: 0 errors, only the 2 pre-existing NU1902; format exit 0) +- [x] run `dotnet test SemiPlot.slnx` (as two foreground project runs: unit 2296/2296 on the second run, + integration 196/196) +- [x] run `dotnet terse` over every touched `.cs` file; it exits 0 (39 files, exit 0) +- ⚠️ the first full unit run failed one test outside this slice, + `PenEditorViewTests.ANameTypedWhileARefreshRuns_ReachesNoFieldAndWritesNothing` ("Expected + name.IsEffectivelyEnabled to be True because the rebuild finished, but found False", run 1 m 35 s); the + class passed 5 of 5 runs alone and the full rerun passed 2296/2296 in 50 s. No PenEditor file is in this + branch; `docs/plans/completed/20260929-window-per-process.md:1183` names the same test as timing-sensitive + +### Task 7: [Final] Update documentation + +**Files:** +- Modify: `docs/architecture/data-integration.md` +- Modify: `docs/architecture/overview.md` +- Modify: `docs/architecture/charting.md` +- Modify: `docs/architecture/trend-interaction.md` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/Postgres/ArchiveStatementTextTests.cs` +- Modify: `AGENTS.md` + +- [x] `data-integration.md:45`: drop `ConnectionFaults` from the interface listing +- [x] `data-integration.md:998-1019`: rename "The connection state the poll reports" to "The connection + state the catalogue read reports" and rewrite it: the producer, the connection class and its SQLSTATEs, + the one rule for the three readers and its three residual cases, the first `Connected` at window open, + and the fault mechanism without a total +- [x] `data-integration.md:1027`, `:1033`, `:1034`, `:1069`, `:1075`, `:1110`, `:1140-1142`: the + `ConnectionLost` row names the catalogue reads; the catalogue and marker rows say a connection-class + failure goes to the bar's entry; the `Unreachable` row lists the widened SQLSTATEs; "because the poll + loop retries" becomes "because the reads retry"; `#no-failure-stops-at-the-log` names the rule and the + three residual cases +- [x] `data-integration.md:1268-1270`: two reads override the backstop, the poll's tick (10 s) and the + catalogue read (5 s); `:1323-1327`: the start sequence's catalogue read fails as `Unreachable` past 5 s, + ahead of the 30 s caller bound +- [x] `overview.md:124-146`: the new build order, the bar over the catalogue sync's stream, the + coordinator paragraph replaced by the subscribe-before-`Start()` rule, and the start sentence (`:146`) + naming the catalogue sync's first read; `:337-338`: the provider no longer reports a connection state +- [x] `overview.md:611-628` (#the-read): one read at window open, the 5 s bound; `:699-704` + (#a-failed-read): the two routes and the state; `:713-716`: the bar takes the sync's stream; + `:731` and `:741-749` (the live markers): a connection-class marker failure goes to the bar's entry +- [x] `charting.md:801-805`: the connection state bullet names `PenCatalogueSync`, not the coordinator +- [x] `trend-interaction.md:818-823`: the catalogue sync no longer reports every 5 s during an outage +- [x] `ArchiveStatementTextTests.cs:219-220`: the tick test's comment no longer names the fault threshold +- [x] `AGENTS.md:360-364`: `PenCatalogueSync` reads once at start too and is the source of the archive + connection state the status bar shows +- [x] `AGENTS.md:376-382`: the failure rule gains the one rule for connection-class failures (they reach the + panel through the bar's `ConnectionLost`) and a pointer to the residual cases in + `data-integration.md#no-failure-stops-at-the-log` +- [x] update every `#the-connection-state-the-poll-reports` link the rename breaks + (`git grep -n "connection-state-the-poll-reports"` prints nothing afterwards) (no tracked doc linked + it; the grep prints only these two plan lines, and the untracked roadmap's link at its line 159 stays + by operator order) + +## Post-Completion + +**Manual verification:** + +1. Start the demo stand: `dotnet run --project SemiPlot/SemiPlot.AppHost`, with a pen drawn in the + minimap. The status bar reads connected and the panel is empty. +2. Stop the stand's bench PostgreSQL container with `docker stop `. Never touch + `scada-capture`. +3. Within 10 to 15 s the indicator reads the fault. The panel holds one `Warning` entry, "The archive stopped + answering", and at most one `Unreachable` entry from the minimap's band read. +4. Wait 30 s. Neither entry gains a repeat or a new timestamp: the loops add nothing, and the minimap logs + its repeats. +5. `docker start` the container. Within 5 s of the server answering the indicator reads connected and one + `Info` entry, "The connection to the archive is restored", appears. +6. Time a stalled server: `docker pause` the bench container and record the seconds to the fault. Write the + measured figure into `data-integration.md` only as a measurement with its date. The roadmap's accepted + figure is about 30 s; a measurement above 40 s goes to the operator for a decision on the bound before it + enters the documentation. + +**Remaining slices** (docs/plans/roadmaps/20261019-opc-live-edge-roadmap.md): + +- live-source-port: a Core live-source port with the poll as its first implementation; + `IDataProvider.Subscribe` leaves `IDataProvider`. +- opc-demo-server: an OPC UA server in the ArchiveSeeder demo writer with the Simple-Scada address space, + port 49875. +- opc-live-source (Closes #116): `SemiPlot.DataSource.OpcUa` feeds the live edge; the poll is deleted. +- opc-settings: an OPC block after the DB block in the settings window. +- live-hold-to-edge: a grey pending gap and a render-only hold of the last value to the edge. + +**Executed by exec:** + +- branch: archive-state-from-catalogue + +Final state at `83301f5`, after the review rounds (Task 6's figures above are from `1de694f`): unit +2297/2297, integration 196/196, `dotnet terse` over 40 touched `.cs` files exit 0, 49 files in +`git diff master...HEAD`. + +## Verify it yourself + +Run from the repository root on `archive-state-from-catalogue`. The integration run needs Docker. + +1. Run the two Acceptance Evidence commands. Expect 160/160 for the unit filter and 20/20 for the + integration filter. +2. Run the guards. Each prints nothing: + `git grep -n "ConnectionFaults" -- SemiPlot` and + `git grep -n "ArchiveConnectionState" -- SemiPlot/SemiPlot.DataSource.Postgres`. +3. One outage, one entry. At `cf21d86` the catalogue loop wrote its third `Unreachable` failure to the + panel and the poll wrote its own `ConnectionLost` beside it. Now + `TrendWindowTests.ThreeUnreachableCatalogueReads_TurnTheBarToFault_WithOneConnectionLostEntry` asserts + one `ConnectionLost` entry, no restamp on a fourth failure, and the `Info` entry on recovery. Task 4 ran + it against the unchanged `TrendWindow.Build` at `338820a`: it failed with `IsConnected` still `true`. + That run is recorded, not repeated here; the two test cases do not exist on `master`. +4. The read at window open. `TrendWindowTests.TheWindowReadsTheCatalogueAtOnce` failed at `338820a` with + `PensQueryCount` 0 and passes now. +5. The catalogue bound. At `cf21d86` `QueryPensAsync` builds its command with no `CommandTimeout`. Now + `ArchiveStatementTextTests.TheCatalogueCommandCarriesItsOwnBoundAndNotTheDataSourceBackstop` pins 5. +6. The connection class. `ArchiveExceptionMapperTests` maps `08006`, `57P01`, `57P02`, `57P03` and `53300` + to `Unreachable` and keeps `57014` as `QueryTimedOut`. At `cf21d86` all five fell through `MapSqlState` + to `ReadFailed`. +7. A missing table is no lost connection. At `cf21d86` + `RealtimePollReadTests.TheFirstSuccessAfterARaisedFaultReportsConnectedExactlyOnce` asserted + `ConnectionLost` for a renamed `trends` table. Now the `PenCatalogueSyncTests` `TableMissing` cases + assert one `TableMissing` entry and no state. +8. On the stand, follow the Manual verification steps above. Expect one behaviour the plan specifies: if the + server returns but answers the catalogue with another kind (`TableMissing`, `AccessDenied`), the + indicator keeps the fault until a read succeeds, beside that kind's own entry + (`ATableMissingReadAfterTheFault_LeavesTheFault_AndReachesThePanelAtOnce`).