diff --git a/AGENTS.md b/AGENTS.md index 9192ef6..5e50876 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -88,9 +88,10 @@ dotnet run --project SemiPlot/SemiPlot.Tools.ArchiveSeeder/SemiPlot.Tools.Archiv --config-dir SemiPlot\Artifacts\dev-config ``` -The demo writer is the same seeder run with `--follow ` instead of `--end`: it appends to -the archive `converge` created and thins it into the coarse layers on every tick -(`docs/architecture/bench.md#the-demo-writer`). +The demo writer is the same seeder run with `--follow --opc-port ` instead of `--end`: +it appends to the archive `converge` created and thins it into the coarse layers on every tick +(`docs/architecture/bench.md#the-demo-writer`), and serves every value over OPC UA on the port +`--opc-port` names (`docs/architecture/bench.md#the-opc-ua-demo-server`). `SemiPlot.AppHost` runs the whole demo stand — the bench container, `converge`, the demo writer and the viewer — in dependency order and stops the processes it launched together; a window the viewer starts @@ -114,7 +115,7 @@ Two projects, split on one axis: crosses a machine boundary (a socket or a conta | Project | Framework | References | Holds | | --- | --- | --- | --- | | `SemiPlot.Tests.Unit` | xunit v3 + `Avalonia.Headless.XUnit` | `SemiPlot.UI`, `SemiPlot.Core`, `SemiPlot.DataSource.Postgres`, `SemiPlot.Tools.ArchiveSeeder` | Every test that opens no socket and starts no container: the UI, the Core models, the seeder's generators and the provider's pure classes | -| `SemiPlot.Tests.Integration` | xunit v3 + `Avalonia.Headless.XUnit` | the same four | The container harness, the container tests and the journeys in one xunit collection, and the socket tests that need no server | +| `SemiPlot.Tests.Integration` | xunit v3 + `Avalonia.Headless.XUnit` | the same four | The container harness, the container tests and the journeys in one xunit collection, the OPC UA demo server's tests in `DemoScadaCollection`, and the socket tests that need no server | - Each project carries its own `TestAppBuilder.cs` with `[assembly: AvaloniaTestApplication]`. Pure logic uses plain `[Fact]`; tests touching ReactiveUI/ScottPlot/Avalonia use @@ -190,11 +191,13 @@ Frame cost during a drag is measured, not asserted: `docs/architecture/testing-s Every test of the `ArchiveDatabaseCollection` in `SemiPlot.Tests.Integration` needs a container runtime and nothing else. The socket classes (`RealtimePollTests`, `CatalogueReadUnreachableTests`, `RealtimeSubscriptionDisposalTests`, `ConvergeReadinessTests`) carry no collection, take no fixture and run without a runtime. +The OPC UA demo server's classes under `DemoScada/` run in `DemoScadaCollection`, which carries no +fixture, and need no runtime either. `SemiPlot/bench/Dockerfile` copies `/semibase` out of `ghcr.io/semiteq/semibase:latest`, which the fixture pulls ahead of the build, and runs `semibase bench` from `/docker-entrypoint-initdb.d/` before the published port opens. The container carries fixed dummy passwords of the fixture's own, and the base image is the constant -`postgres:17-alpine`. A missing runtime fails every test of the collection with +`postgres:17-alpine`. A missing runtime fails every test of `ArchiveDatabaseCollection` with `TestPipelineException`; nothing skips. A run leaves nothing behind but the images it pulled: the resource reaper removes the built image, diff --git a/SemiPlot/Directory.Packages.props b/SemiPlot/Directory.Packages.props index c8d585b..2193571 100644 --- a/SemiPlot/Directory.Packages.props +++ b/SemiPlot/Directory.Packages.props @@ -16,14 +16,17 @@ - - + + - - + + + + + diff --git a/SemiPlot/SemiPlot.AppHost/AppHost.cs b/SemiPlot/SemiPlot.AppHost/AppHost.cs index 563bb1a..c76f740 100644 --- a/SemiPlot/SemiPlot.AppHost/AppHost.cs +++ b/SemiPlot/SemiPlot.AppHost/AppHost.cs @@ -19,6 +19,8 @@ const ushort ContainerPort = 5432; // One density for the seeded day and the live tail, so the chart shows no seam between them. const string ChangeSeconds = "0.5"; +// A running Simple-Scada holds 49875, so the stand serves OPC UA on a port of its own. +const string OpcPort = "55875"; // AppHostDirectory is SemiPlot/SemiPlot.AppHost, so the tracked set sits two levels above it. var trackedConfiguration = Path.GetFullPath(Path.Combine(builder.AppHostDirectory, "..", "..", "ConfigFiles")); @@ -48,7 +50,11 @@ .WaitFor(bench); builder.AddProject("writer") - .WithArgs("--connection", writerConnection, "--follow", "1", "--change-seconds", ChangeSeconds) + .WithArgs( + "--connection", writerConnection, + "--follow", "1", + "--change-seconds", ChangeSeconds, + "--opc-port", OpcPort) .WaitForCompletion(converge); builder.AddProject("viewer") diff --git a/SemiPlot/SemiPlot.Tests.Integration/CoarseFlushTests.cs b/SemiPlot/SemiPlot.Tests.Integration/CoarseFlushTests.cs index 9123ac4..4bca9d3 100644 --- a/SemiPlot/SemiPlot.Tests.Integration/CoarseFlushTests.cs +++ b/SemiPlot/SemiPlot.Tests.Integration/CoarseFlushTests.cs @@ -248,7 +248,9 @@ private FollowOptions Options() TimeSpan.FromSeconds(1), PenCount, SeederOptions.DefaultSeed, - SeederOptions.DefaultChangeSeconds); + SeederOptions.DefaultChangeSeconds, + TimeSpan.Zero, + 55875); } // What LayerThinner selects for one period, in the order the read returns. diff --git a/SemiPlot/SemiPlot.Tests.Integration/DemoScada/DemoScadaCollection.cs b/SemiPlot/SemiPlot.Tests.Integration/DemoScada/DemoScadaCollection.cs new file mode 100644 index 0000000..00e8764 --- /dev/null +++ b/SemiPlot/SemiPlot.Tests.Integration/DemoScada/DemoScadaCollection.cs @@ -0,0 +1,10 @@ +using Xunit; + +namespace SemiPlot.Tests.Integration.DemoScada; + +// Every demo-server test joins this collection so no two servers start side by side. +[CollectionDefinition(Name)] +public sealed class DemoScadaCollection +{ + public const string Name = "demo-scada"; +} diff --git a/SemiPlot/SemiPlot.Tests.Integration/DemoScada/DemoScadaHost.cs b/SemiPlot/SemiPlot.Tests.Integration/DemoScada/DemoScadaHost.cs new file mode 100644 index 0000000..75e42d1 --- /dev/null +++ b/SemiPlot/SemiPlot.Tests.Integration/DemoScada/DemoScadaHost.cs @@ -0,0 +1,114 @@ +using System.Net; +using System.Net.Sockets; + +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Testing; + +using Opc.Ua; + +using SemiPlot.Tools.ArchiveSeeder; + +namespace SemiPlot.Tests.Integration.DemoScada; + +/// Starts demo servers over one temporary certificate directory and one log collector. +public sealed class DemoScadaHost : IDisposable +{ + private const int StartAttempts = 3; + + // 4840 is where the stack binds when asked for port 0, and 49875 is the real Simple-Scada's port. + private static readonly int[] _reservedPorts = [4840, 49875]; + + private readonly ILoggerFactory _loggerFactory; + + public DemoScadaHost() + { + CertificateDirectory = Path.Combine( + Path.GetTempPath(), + "SemiPlot.Tests", + $"DemoScada-{Guid.NewGuid():N}"); + _loggerFactory = LoggerFactory.Create(builder => builder + .AddProvider(new FakeLoggerProvider(Logs)) + .SetMinimumLevel(LogLevel.Warning)); + } + + /// The zone every test server archives in: no DST, and nine hours off a UTC test machine. + public static TimeZoneInfo ArchiveZone { get; } = TimeZoneInfo.FindSystemTimeZoneById("Asia/Tokyo"); + + /// picks the port. + public static FollowOptions Options { get; } = new( + "Host=127.0.0.1;Database=unused", + TimeSpan.FromSeconds(1), + PenCount: 8, + Seed: 1, + ChangeSeconds: 0.5, + ArchiveLag: TimeSpan.Zero, + OpcPort: 0); + + public string CertificateDirectory { get; } + + public FakeLogCollector Logs { get; } = new(); + + /// Starts a server on a free loopback port, retrying when another process takes the port first. + public async Task StartAsync(FollowOptions options) + { + for (var attempt = 1; ; attempt++) + { + try + { + return await StartOnPortAsync(options with { OpcPort = FreePort() }); + } + catch (SeederException exception) when (attempt < StartAttempts && IsBusyPort(exception)) + { + } + } + } + + private static bool IsBusyPort(SeederException exception) + { + return exception.InnerException is ServiceResultException listener + && listener.StatusCode == StatusCodes.BadNoCommunication; + } + + public Task StartOnPortAsync(FollowOptions options) + { + return DemoScadaServer.StartAsync(options, ArchiveZone, CertificateDirectory, _loggerFactory); + } + + public static int FreePort() + { + while (true) + { + var listener = new TcpListener(IPAddress.Loopback, 0); + listener.Start(); + var port = ((IPEndPoint)listener.LocalEndpoint).Port; + listener.Stop(); + + if (!_reservedPorts.Contains(port)) + { + return port; + } + } + } + + /// The server's records at Warning and above so far, for an assertion's reason. + public string ServerLog() + { + var records = Logs.GetSnapshot(); + + return records.Count == 0 + ? "the server logged nothing at Warning or above" + : "the server logged " + string.Join( + Environment.NewLine, + records.Select(record => $"{record.Level} {record.Category}: {record.Message} {record.Exception}")); + } + + public void Dispose() + { + _loggerFactory.Dispose(); + + if (Directory.Exists(CertificateDirectory)) + { + Directory.Delete(CertificateDirectory, recursive: true); + } + } +} diff --git a/SemiPlot/SemiPlot.Tests.Integration/DemoScada/DemoScadaServerLifecycleTests.cs b/SemiPlot/SemiPlot.Tests.Integration/DemoScada/DemoScadaServerLifecycleTests.cs new file mode 100644 index 0000000..f25eccc --- /dev/null +++ b/SemiPlot/SemiPlot.Tests.Integration/DemoScada/DemoScadaServerLifecycleTests.cs @@ -0,0 +1,242 @@ +using System.Net; +using System.Net.Sockets; + +using AwesomeAssertions; + +using Opc.Ua; + +using SemiPlot.DataSource.Postgres; +using SemiPlot.Tools.ArchiveSeeder; + +using Xunit; + +using static SemiPlot.Tests.Integration.DemoScada.SimpleScadaNodeIds; + +namespace SemiPlot.Tests.Integration.DemoScada; + +[Collection(DemoScadaCollection.Name)] +[Trait("Component", "Core")] +[Trait("Area", "Data")] +[Trait("Category", "Integration")] +public sealed class DemoScadaServerLifecycleTests : IDisposable +{ + private const int TestTimeoutMilliseconds = 120000; + private const string NoneSecurityPolicyUri = "http://opcfoundation.org/UA/SecurityPolicy#None"; + private const string Basic256Sha256SecurityPolicyUri = "http://opcfoundation.org/UA/SecurityPolicy#Basic256Sha256"; + private const double OneChangePerDaySeconds = 86400; + private const double ChangeSecondsPastTheTaskDelayLimit = 100000.0 * 86400; + + private static readonly TimeSpan _changeFreeBudget = TimeSpan.FromSeconds(30); + + private readonly DemoScadaHost _host = new(); + + [Fact(Timeout = TestTimeoutMilliseconds)] + public async Task TheServerAdvertisesNoneSignAndSignAndEncrypt_EachWithOneAnonymousToken() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var server = await _host.StartAsync(DemoScadaHost.Options); + + var endpoints = await OpcTestClient.GetEndpointsAsync(server.EndpointUrl, cancellationToken); + + endpoints.Select(endpoint => (endpoint.SecurityMode, endpoint.SecurityPolicyUri)).Should().BeEquivalentTo( + [ + (MessageSecurityMode.None, NoneSecurityPolicyUri), + (MessageSecurityMode.Sign, Basic256Sha256SecurityPolicyUri), + (MessageSecurityMode.SignAndEncrypt, Basic256Sha256SecurityPolicyUri), + ], + _host.ServerLog()); + + endpoints.Should().AllSatisfy( + endpoint => endpoint.UserIdentityTokens.ToArray().Should().ContainSingle() + .Which.TokenType.Should().Be(UserTokenType.Anonymous), + _host.ServerLog()); + } + + [Fact(Timeout = TestTimeoutMilliseconds)] + public async Task TheNoneEndpointAdvertisesItsAnonymousTokenOnTheNonePolicy() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var server = await _host.StartAsync(DemoScadaHost.Options); + + var endpoints = await OpcTestClient.GetEndpointsAsync(server.EndpointUrl, cancellationToken); + + endpoints.Single(endpoint => endpoint.SecurityMode == MessageSecurityMode.None) + .UserIdentityTokens.ToArray().Should().ContainSingle() + .Which.SecurityPolicyUri.Should().Be( + NoneSecurityPolicyUri, + "the real Simple-Scada advertises its anonymous token on #None; {0}", + _host.ServerLog()); + } + + [Fact(Timeout = TestTimeoutMilliseconds)] + public async Task ASessionOpensOverNoneWithTheAdvertisedAnonymousToken() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var server = await _host.StartAsync(DemoScadaHost.Options); + + await using var client = await OpcTestClient.ConnectAsync(server.EndpointUrl, cancellationToken); + + client.Connected.Should().BeTrue(_host.ServerLog()); + } + + [Fact(Timeout = TestTimeoutMilliseconds)] + public async Task TheSimpleScadaNamespaceSitsAtIndexThree() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var server = await _host.StartAsync(DemoScadaHost.Options); + + await using var client = await OpcTestClient.ConnectAsync(server.EndpointUrl, cancellationToken); + + client.NamespaceUris.GetIndex(NamespaceUri).Should().Be(NamespaceIndex, _host.ServerLog()); + } + + [Fact(Timeout = TestTimeoutMilliseconds)] + public async Task APortHeldOnLoopback_FailsTheStartNamingThePort() + { + using var holder = new TcpListener(IPAddress.Loopback, DemoScadaHost.FreePort()); + holder.Start(); + var port = ((IPEndPoint)holder.LocalEndpoint).Port; + + var start = () => _host.StartOnPortAsync(DemoScadaHost.Options with { OpcPort = port }); + + (await start.Should().ThrowAsync(_host.ServerLog())) + .WithMessage($"*OPC UA port {port} is in use*"); + } + + [Fact(Timeout = TestTimeoutMilliseconds)] + public async Task AServerStoppedWithASessionAttached_StartsAgainOnItsPort() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var first = await _host.StartAsync(DemoScadaHost.Options); + var port = new Uri(first.EndpointUrl).Port; + await using var abandoned = await OpcTestClient.ConnectAsync(first.EndpointUrl, cancellationToken); + + await first.DisposeAsync(); + await using var second = await _host.StartOnPortAsync(DemoScadaHost.Options with { OpcPort = port }); + await using var client = await OpcTestClient.ConnectAsync(second.EndpointUrl, cancellationToken); + + second.EndpointUrl.Should().Be(first.EndpointUrl); + client.Connected.Should().BeTrue(_host.ServerLog()); + } + + [Fact(Timeout = TestTimeoutMilliseconds)] + public async Task ASecondStartInTheSameCertificateDirectory_ReusesTheCertificate() + { + var certificates = Path.Combine(_host.CertificateDirectory, "own", "certs"); + await using (await _host.StartAsync(DemoScadaHost.Options)) + { + } + + var firstCertificate = Directory.GetFiles(certificates).Should().ContainSingle(_host.ServerLog()).Which; + var firstBytes = await File.ReadAllBytesAsync(firstCertificate, TestContext.Current.CancellationToken); + + await using var second = await _host.StartAsync(DemoScadaHost.Options); + + var secondCertificate = Directory.GetFiles(certificates).Should().ContainSingle(_host.ServerLog()).Which; + secondCertificate.Should().Be(firstCertificate, _host.ServerLog()); + (await File.ReadAllBytesAsync(secondCertificate, TestContext.Current.CancellationToken)) + .Should().Equal(firstBytes, _host.ServerLog()); + } + + [Fact(Timeout = TestTimeoutMilliseconds)] + public async Task Completion_RunsUntilDisposeAsync_AndThenCompletesSuccessfully() + { + await using var server = await _host.StartAsync(DemoScadaHost.Options); + + server.Completion.IsCompleted.Should().BeFalse(_host.ServerLog()); + + await server.Invoking(started => started.DisposeAsync().AsTask()).Should().NotThrowAsync(_host.ServerLog()); + server.Completion.IsCompletedSuccessfully.Should().BeTrue(_host.ServerLog()); + } + + [Fact(Timeout = TestTimeoutMilliseconds)] + public async Task AFaultedPublishingLoop_FaultsCompletion_AndDisposeAsyncRethrowsTheFault() + { + await using var server = await _host.StartAsync( + DemoScadaHost.Options with { ChangeSeconds = ChangeSecondsPastTheTaskDelayLimit }); + + var loopFault = (await server.Awaiting(started => started.Completion) + .Should().ThrowAsync(_host.ServerLog())).Which; + + (await server.Invoking(started => started.DisposeAsync().AsTask()) + .Should().ThrowAsync(_host.ServerLog())).Which.Should().BeSameAs(loopFault, _host.ServerLog()); + } + + [Fact(Timeout = TestTimeoutMilliseconds)] + public async Task EveryFirstNotificationAfterAStart_CarriesTheStartInstantAndTheLatticeValueThere() + { + var cancellationToken = TestContext.Current.CancellationToken; + var options = DemoScadaHost.Options with { ChangeSeconds = OneChangePerDaySeconds }; + var converter = new ArchiveTimeConverter(DemoScadaHost.ArchiveZone); + var pens = RawLayerGenerator.SelectPens(options.PenCount); + var nextChangeUtc = await WaitUntilNoChangeIsNearAsync(options, converter, cancellationToken); + + var beforeStart = DateTime.UtcNow; + await using var server = await _host.StartAsync(options); + var afterStart = DateTime.UtcNow; + await using var client = await OpcTestClient.ConnectAsync(server.EndpointUrl, cancellationToken); + var subscription = await client.SubscribeAsync( + [.. pens.Select(pen => PenNode(pen.PenId))], + cancellationToken); + var firsts = new Dictionary(); + + await foreach (var notification in subscription.Notifications.ReadAllAsync(cancellationToken)) + { + firsts.TryAdd(notification.Node, notification); + + if (firsts.Count == pens.Count) + { + break; + } + } + + DateTime.UtcNow.Should().BeBefore(nextChangeUtc, "the test window crossed a lattice change; raise the budget"); + + foreach (var pen in pens) + { + var first = firsts[PenNode(pen.PenId)].Value; + var stamp = first.SourceTimestamp.ToDateTime(); + var expected = LiveTailGenerator.ChangesAt(options, converter.ToArchiveLocal(stamp)) + .Single(row => row.Id == pen.PenId); + + stamp.Should().BeOnOrAfter(beforeStart, "pen {0}; {1}", pen.PenId, _host.ServerLog()) + .And.BeOnOrBefore(afterStart, "pen {0}; {1}", pen.PenId, _host.ServerLog()); + first.WrappedValue.GetDouble(double.NaN).Should().Be( + expected.Value, + "pen {0}; {1}", + pen.PenId, + _host.ServerLog()); + } + } + + public void Dispose() + { + _host.Dispose(); + } + + /// Waits past a lattice change due within the budget and returns the next change in UTC. + private static async Task WaitUntilNoChangeIsNearAsync( + FollowOptions options, + ArchiveTimeConverter converter, + CancellationToken cancellationToken) + { + var nextChangeUtc = NextChangeUtc(options, converter); + var untilNextChange = nextChangeUtc - DateTime.UtcNow; + + if (untilNextChange >= _changeFreeBudget) + { + return nextChangeUtc; + } + + await Task.Delay(untilNextChange + TimeSpan.FromSeconds(1), cancellationToken); + + return NextChangeUtc(options, converter); + } + + private static DateTime NextChangeUtc(FollowOptions options, ArchiveTimeConverter converter) + { + var latestChange = LiveTailGenerator.ChangesAt(options, converter.ToArchiveLocal(DateTime.UtcNow))[0]; + + return converter.ToUtc(latestChange.Timestamp.AddSeconds(options.ChangeSeconds)); + } +} diff --git a/SemiPlot/SemiPlot.Tests.Integration/DemoScada/DemoScadaServerTests.cs b/SemiPlot/SemiPlot.Tests.Integration/DemoScada/DemoScadaServerTests.cs new file mode 100644 index 0000000..7f2718d --- /dev/null +++ b/SemiPlot/SemiPlot.Tests.Integration/DemoScada/DemoScadaServerTests.cs @@ -0,0 +1,266 @@ +using System.Threading.Channels; + +using AwesomeAssertions; + +using Opc.Ua; + +using SemiPlot.DataSource.Postgres; +using SemiPlot.Tools.ArchiveSeeder; + +using Xunit; + +using static SemiPlot.Tests.Integration.DemoScada.SimpleScadaNodeIds; + +namespace SemiPlot.Tests.Integration.DemoScada; + +[Collection(DemoScadaCollection.Name)] +[Trait("Component", "Core")] +[Trait("Area", "Data")] +[Trait("Category", "Integration")] +public sealed class DemoScadaServerTests : IClassFixture +{ + private const int TestTimeoutMilliseconds = 120000; + private const int UnknownPenId = 99999; + private const string WatchlistGroup = "Watchlist"; + + private static readonly TimeSpan _observationWindow = TimeSpan.FromSeconds(3); + private static readonly TimeSpan _clockTolerance = TimeSpan.FromSeconds(60); + + private readonly DemoScadaServerFixture _fixture; + + public DemoScadaServerTests(DemoScadaServerFixture fixture) + { + _fixture = fixture; + _fixture.Host.Logs.Clear(); + } + + private static IReadOnlyList Pens => + RawLayerGenerator.SelectPens(DemoScadaServerFixture.Options.PenCount); + + [Fact(Timeout = TestTimeoutMilliseconds)] + public async Task Objects_ListsOneRootAheadOfTheGroupFoldersNumberedFromThree() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var client = await OpcTestClient.ConnectAsync(_fixture.Server.EndpointUrl, cancellationToken); + + var folders = (await client.BrowseAsync(ObjectIds.ObjectsFolder, cancellationToken)) + .Where(reference => reference.BrowseName.NamespaceIndex == NamespaceIndex) + .Select(reference => ( + Node: client.ToNodeId(reference.NodeId), + Name: reference.BrowseName.Name ?? string.Empty)) + .ToArray(); + var groups = Pens + .Select(pen => pen.PrimaryGroup) + .Where(group => group.Length > 0) + .Distinct(StringComparer.Ordinal) + .ToArray(); + var expected = groups + .Select((group, position) => (Node: FolderNode(group, FirstGroupFolderNumber + position), Name: group)) + .Prepend((Node: FolderNode(RootFolderName, RootFolderNumber), Name: RootFolderName)); + + folders.Should().Equal(expected, _fixture.Host.ServerLog()); + folders.Should().ContainSingle(folder => folder.Name == RootFolderName, _fixture.Host.ServerLog()); + } + + [Fact(Timeout = TestTimeoutMilliseconds)] + public async Task Root_OrganizesEveryPenUnderItsIdWithItsCatalogueNameAsADouble() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var client = await OpcTestClient.ConnectAsync(_fixture.Server.EndpointUrl, cancellationToken); + + var variables = await BrowseVariablesAsync( + client, + FolderNode(RootFolderName, RootFolderNumber), + cancellationToken); + + variables.Should().BeEquivalentTo( + Pens.Select(pen => (Node: PenNode(pen.PenId), pen.Name)), + options => options.ComparingByValue(), + _fixture.Host.ServerLog()); + + foreach (var pen in Pens) + { + var node = await client.ReadNodeAsync(PenNode(pen.PenId), cancellationToken); + + node.Should().BeOfType(_fixture.Host.ServerLog()) + .Which.DataType.Should().Be(DataTypeIds.Double, _fixture.Host.ServerLog()); + } + } + + [Fact(Timeout = TestTimeoutMilliseconds)] + public async Task EachGroupFolder_OrganizesItsPrimaryGroupPensUnderTheSameIds() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var client = await OpcTestClient.ConnectAsync(_fixture.Server.EndpointUrl, cancellationToken); + var groups = Pens + .Where(pen => pen.PrimaryGroup.Length > 0) + .GroupBy(pen => pen.PrimaryGroup, StringComparer.Ordinal) + .ToArray(); + + for (var position = 0; position < groups.Length; position++) + { + var group = groups[position]; + var folder = FolderNode(group.Key, FirstGroupFolderNumber + position); + + var variables = await BrowseVariablesAsync(client, folder, cancellationToken); + + variables.Should().BeEquivalentTo( + group.Select(pen => (Node: PenNode(pen.PenId), pen.Name)), + options => options.ComparingByValue(), + "folder {0}; {1}", + group.Key, + _fixture.Host.ServerLog()); + } + } + + [Fact(Timeout = TestTimeoutMilliseconds)] + public async Task TheUngroupedPenSitsUnderRootOnly_AndALaterGroupGetsNoFolder() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var client = await OpcTestClient.ConnectAsync(_fixture.Server.EndpointUrl, cancellationToken); + var ungrouped = PenNode(SyntheticPenCatalog.UncommissionedPenId); + + var folders = (await client.BrowseAsync(ObjectIds.ObjectsFolder, cancellationToken)) + .Where(reference => reference.BrowseName.NamespaceIndex == NamespaceIndex) + .ToArray(); + var holders = new List(); + + foreach (var folder in folders) + { + var variables = await BrowseVariablesAsync(client, client.ToNodeId(folder.NodeId), cancellationToken); + + if (variables.Any(variable => variable.Node == ungrouped)) + { + holders.Add(folder.BrowseName.Name ?? string.Empty); + } + } + + Pens.Should().Contain(pen => pen.PenId == SyntheticPenCatalog.UncommissionedPenId); + Pens.Should().Contain(pen => pen.Groups.Contains(WatchlistGroup)); + holders.Should().Equal([RootFolderName], _fixture.Host.ServerLog()); + folders.Should().NotContain(folder => folder.BrowseName.Name == WatchlistGroup, _fixture.Host.ServerLog()); + } + + [Fact(Timeout = TestTimeoutMilliseconds)] + public async Task EveryChangeCarriesTheLatticeValueAtItsTrueUtcInstant() + { + var cancellationToken = TestContext.Current.CancellationToken; + var options = DemoScadaServerFixture.Options; + var converter = new ArchiveTimeConverter(DemoScadaHost.ArchiveZone); + var intervalTicks = TimeSpan.FromSeconds(options.ChangeSeconds).Ticks; + await using var client = await OpcTestClient.ConnectAsync(_fixture.Server.EndpointUrl, cancellationToken); + var subscription = await client.SubscribeAsync( + [.. Pens.Select(pen => PenNode(pen.PenId))], + cancellationToken); + + await Task.Delay(_observationWindow, cancellationToken); + var notifications = Drain(subscription.Notifications); + + foreach (var pen in Pens) + { + var received = notifications.Where(notification => notification.Node == PenNode(pen.PenId)).ToArray(); + var stamps = received.Select(notification => notification.Value.SourceTimestamp.ToDateTime()).ToArray(); + + received.Should().HaveCountGreaterThanOrEqualTo(3, "pen {0}; {1}", pen.PenId, _fixture.Host.ServerLog()); + stamps.Should().BeInAscendingOrder("pen {0}; {1}", pen.PenId, _fixture.Host.ServerLog()) + .And.OnlyHaveUniqueItems("pen {0}; {1}", pen.PenId, _fixture.Host.ServerLog()); + + foreach (var change in received.Skip(1)) + { + var stamp = change.Value.SourceTimestamp.ToDateTime(); + var local = converter.ToArchiveLocal(stamp); + var expected = LiveTailGenerator.ChangesAt(options, local).Single(row => row.Id == pen.PenId); + + (local.Ticks % intervalTicks).Should().Be( + 0, + "pen {0} at {1:O} is no lattice instant; {2}", + pen.PenId, + local, + _fixture.Host.ServerLog()); + change.Value.WrappedValue.GetDouble(double.NaN).Should().Be( + expected.Value, + "pen {0} at {1:O}; {2}", + pen.PenId, + local, + _fixture.Host.ServerLog()); + stamp.Should().BeCloseTo( + change.ReceivedUtc, + _clockTolerance, + "a local stamp sent as UTC misses by the zone's offset; {0}", + _fixture.Host.ServerLog()); + } + } + } + + [Fact(Timeout = TestTimeoutMilliseconds)] + public async Task AnUnknownNode_FailsItsMonitoredItemWhileThePensAreCreated() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var client = await OpcTestClient.ConnectAsync(_fixture.Server.EndpointUrl, cancellationToken); + var unknown = PenNode(UnknownPenId); + + var subscription = await client.SubscribeAsync( + [.. Pens.Select(pen => PenNode(pen.PenId)), unknown], + cancellationToken); + + subscription.ItemStatuses[unknown].Should().Be(StatusCodes.BadNodeIdUnknown, _fixture.Host.ServerLog()); + subscription.ItemStatuses.Where(item => item.Key != unknown).Should().AllSatisfy( + item => item.Value.Should().Be(StatusCodes.Good), + _fixture.Host.ServerLog()); + } + + private static async Task> BrowseVariablesAsync( + OpcTestClient client, + NodeId folder, + CancellationToken cancellationToken) + { + var references = await client.BrowseAsync(folder, cancellationToken); + + return + [ + .. references + .Where(reference => reference.NodeClass == NodeClass.Variable) + .Select(reference => (client.ToNodeId(reference.NodeId), reference.BrowseName.Name ?? string.Empty)), + ]; + } + + private static List Drain(ChannelReader notifications) + { + var drained = new List(); + + while (notifications.TryRead(out var notification)) + { + drained.Add(notification); + } + + return drained; + } +} + +/// One demo server serving the stand's eight pens on a fast lattice, shared by a test class. +public sealed class DemoScadaServerFixture : IAsyncLifetime +{ + public static readonly FollowOptions Options = DemoScadaHost.Options with { ChangeSeconds = 0.25 }; + + private DemoScadaServer? _server; + + public DemoScadaHost Host { get; } = new(); + + public DemoScadaServer Server => + _server ?? throw new InvalidOperationException("The demo server was used before it was started."); + + public async ValueTask InitializeAsync() + { + _server = await Host.StartAsync(Options); + } + + public async ValueTask DisposeAsync() + { + if (_server is not null) + { + await _server.DisposeAsync(); + } + + Host.Dispose(); + } +} diff --git a/SemiPlot/SemiPlot.Tests.Integration/DemoScada/OpcTestClient.cs b/SemiPlot/SemiPlot.Tests.Integration/DemoScada/OpcTestClient.cs new file mode 100644 index 0000000..a0a1e6c --- /dev/null +++ b/SemiPlot/SemiPlot.Tests.Integration/DemoScada/OpcTestClient.cs @@ -0,0 +1,175 @@ +using System.Threading.Channels; + +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; + +using Opc.Ua; +using Opc.Ua.Client; + +using MonitoredItem = Opc.Ua.Client.MonitoredItem; +using Subscription = Opc.Ua.Client.Subscription; + +namespace SemiPlot.Tests.Integration.DemoScada; + +/// An anonymous client over the None endpoint with no application certificate. +public sealed class OpcTestClient : IAsyncDisposable +{ + private const uint SessionTimeoutMilliseconds = 60000; + private const int PublishingIntervalMilliseconds = 100; + private const int SamplingIntervalMilliseconds = 100; + + // A session whose server has gone answers no CloseSession; the client gives up after this. + private static readonly TimeSpan _closeTimeout = TimeSpan.FromSeconds(2); + + private static readonly ITelemetryContext _telemetry = new Telemetry(NullLoggerFactory.Instance); + + private readonly ISession _session; + + private OpcTestClient(ISession session) + { + _session = session; + } + + public bool Connected => _session.Connected; + + public NamespaceTable NamespaceUris => _session.NamespaceUris; + + public static async Task> GetEndpointsAsync( + string endpointUrl, + CancellationToken cancellationToken) + { + using var discovery = await DiscoveryClient.CreateAsync( + CreateConfiguration(), + new Uri(endpointUrl), + DiagnosticsMasks.None, + cancellationToken); + + return [.. await discovery.GetEndpointsAsync(default, cancellationToken)]; + } + + /// Opens a session with the anonymous token policy the server's None endpoint advertises. + public static async Task ConnectAsync(string endpointUrl, CancellationToken cancellationToken) + { + var configuration = CreateConfiguration(); + var endpoints = await GetEndpointsAsync(endpointUrl, cancellationToken); + var none = endpoints.Single(endpoint => endpoint.SecurityMode == MessageSecurityMode.None + && endpoint.SecurityPolicyUri == SecurityPolicies.None); + var configured = new ConfiguredEndpoint(null, none, EndpointConfiguration.Create(configuration)); + var session = await new DefaultSessionFactory(_telemetry).CreateAsync( + configuration, + configured, + false, + false, + nameof(OpcTestClient), + SessionTimeoutMilliseconds, + new UserIdentity(), + default, + cancellationToken); + + return new OpcTestClient(session); + } + + public async Task> BrowseAsync( + NodeId node, + CancellationToken cancellationToken) + { + var browser = new Browser(_session) + { + BrowseDirection = BrowseDirection.Forward, + ReferenceTypeId = ReferenceTypeIds.HierarchicalReferences, + IncludeSubtypes = true, + NodeClassMask = (int)(NodeClass.Object | NodeClass.Variable), + }; + + return [.. await browser.BrowseAsync(node, cancellationToken)]; + } + + public NodeId ToNodeId(ExpandedNodeId node) + { + return ExpandedNodeId.ToNodeId(node, _session.NamespaceUris); + } + + public Task ReadNodeAsync(NodeId node, CancellationToken cancellationToken) + { + return _session.ReadNodeAsync(node, cancellationToken); + } + + /// Monitors the nodes with a queue of one and collects every data change into a channel. + public async Task SubscribeAsync( + IReadOnlyList nodes, + CancellationToken cancellationToken) + { + var notifications = Channel.CreateUnbounded(); + var subscription = new Subscription(_session.DefaultSubscription) + { + PublishingInterval = PublishingIntervalMilliseconds, + TimestampsToReturn = TimestampsToReturn.Both, + }; + + foreach (var node in nodes) + { + var item = new MonitoredItem(subscription.DefaultItem) + { + StartNodeId = node, + SamplingInterval = SamplingIntervalMilliseconds, + QueueSize = 1, + }; + item.Notification += (monitored, arguments) => + { + if (arguments.NotificationValue is MonitoredItemNotification notification) + { + notifications.Writer.TryWrite( + new OpcNotification(monitored.StartNodeId, notification.Value, DateTime.UtcNow)); + } + }; + subscription.AddItem(item); + } + + _session.AddSubscription(subscription); + await subscription.CreateAsync(cancellationToken); + + var statuses = subscription.MonitoredItems.ToDictionary( + item => item.StartNodeId, + item => item.Status.Error?.StatusCode ?? StatusCodes.Good); + + return new OpcTestSubscription(notifications.Reader, statuses); + } + + public async ValueTask DisposeAsync() + { + using var giveUp = new CancellationTokenSource(_closeTimeout); + + try + { + await _session.CloseAsync(giveUp.Token); + } + catch (Exception exception) when (exception is ServiceResultException or OperationCanceledException) + { + } + + _session.Dispose(); + } + + private static ApplicationConfiguration CreateConfiguration() + { + return new ApplicationConfiguration(_telemetry) + { + ApplicationName = nameof(OpcTestClient), + ApplicationUri = $"urn:localhost:{nameof(OpcTestClient)}", + ApplicationType = ApplicationType.Client, + SecurityConfiguration = new SecurityConfiguration(), + TransportQuotas = new TransportQuotas(), + ClientConfiguration = new ClientConfiguration { DefaultSessionTimeout = (int)SessionTimeoutMilliseconds }, + }; + } + + private sealed class Telemetry(ILoggerFactory loggerFactory) : TelemetryContextBase(loggerFactory); +} + +/// One data change as the client received it. +public sealed record OpcNotification(NodeId Node, DataValue Value, DateTime ReceivedUtc); + +/// The notifications of one subscription and the status each monitored item was created with. +public sealed record OpcTestSubscription( + ChannelReader Notifications, + IReadOnlyDictionary ItemStatuses); diff --git a/SemiPlot/SemiPlot.Tests.Integration/DemoScada/SimpleScadaNodeIds.cs b/SemiPlot/SemiPlot.Tests.Integration/DemoScada/SimpleScadaNodeIds.cs new file mode 100644 index 0000000..55dd600 --- /dev/null +++ b/SemiPlot/SemiPlot.Tests.Integration/DemoScada/SimpleScadaNodeIds.cs @@ -0,0 +1,25 @@ +using Opc.Ua; + +namespace SemiPlot.Tests.Integration.DemoScada; + +/// The node ids the demo server serves, spelled the way Simple-Scada spells its own. +public static class SimpleScadaNodeIds +{ + public const string NamespaceUri = "http://simple-scada.com/UA"; + public const ushort NamespaceIndex = 3; + public const string RootFolderName = "root"; + public const int RootFolderNumber = 2; + public const int FirstGroupFolderNumber = 3; + + private const string ProjectId = "5E4D0C1A0B"; + + public static NodeId PenNode(int penId) + { + return new NodeId($"{ProjectId}{penId}", NamespaceIndex); + } + + public static NodeId FolderNode(string name, int number) + { + return new NodeId($"{ProjectId}{name}_{number}", NamespaceIndex); + } +} diff --git a/SemiPlot/SemiPlot.Tests.Integration/FollowRestartTests.cs b/SemiPlot/SemiPlot.Tests.Integration/FollowRestartTests.cs index f7db5f9..2726e47 100644 --- a/SemiPlot/SemiPlot.Tests.Integration/FollowRestartTests.cs +++ b/SemiPlot/SemiPlot.Tests.Integration/FollowRestartTests.cs @@ -79,7 +79,9 @@ await Writer().WriteAsync( private static FollowOptions Options() { - return new("Host=127.0.0.1;Database=archive", TimeSpan.FromSeconds(1), PenCount, 1L, ChangeSeconds); + return new( + "Host=127.0.0.1;Database=archive", TimeSpan.FromSeconds(1), PenCount, 1L, ChangeSeconds, + TimeSpan.Zero, 55875); } private Task CountRawRowsAsync(CancellationToken cancellationToken) diff --git a/SemiPlot/SemiPlot.Tests.Integration/SemiPlot.Tests.Integration.csproj b/SemiPlot/SemiPlot.Tests.Integration/SemiPlot.Tests.Integration.csproj index b64f228..6bbb1bc 100644 --- a/SemiPlot/SemiPlot.Tests.Integration/SemiPlot.Tests.Integration.csproj +++ b/SemiPlot/SemiPlot.Tests.Integration/SemiPlot.Tests.Integration.csproj @@ -11,6 +11,7 @@ + diff --git a/SemiPlot/SemiPlot.Tests.Integration/StaleArchiveGuardTests.cs b/SemiPlot/SemiPlot.Tests.Integration/StaleArchiveGuardTests.cs index 1ef89a0..2798287 100644 --- a/SemiPlot/SemiPlot.Tests.Integration/StaleArchiveGuardTests.cs +++ b/SemiPlot/SemiPlot.Tests.Integration/StaleArchiveGuardTests.cs @@ -19,10 +19,9 @@ public sealed class StaleArchiveGuardTests(PostgresContainerFixture postgresCont private static readonly DateTime _now = new(2026, 1, 2, 12, 0, 0, DateTimeKind.Unspecified); - // The state the operator reached: a fill from the previous day under a writer started today. The - // refusal names the verb that refills, because that is the whole remedy. + // The state the operator reached: a fill from the previous day under a writer started today. [Fact] - public async Task AnArchiveOlderThanTheBoundIsRefusedAndNamesConverge() + public async Task AnArchiveOlderThanTheBoundIsRefusedNamingConvergeAndTheArchiveLag() { var newest = _now.AddMinutes(-793.7); @@ -33,7 +32,7 @@ public async Task AnArchiveOlderThanTheBoundIsRefusedAndNamesConverge() var refused = (await act.Should().ThrowAsync()).Which; - refused.Message.Should().Contain("converge"); + refused.Message.Should().Contain("converge").And.Contain("--archive-lag"); } // The caller reads the absent timestamp as "start at the clock". diff --git a/SemiPlot/SemiPlot.Tests.Unit/FollowEdgeTests.cs b/SemiPlot/SemiPlot.Tests.Unit/FollowEdgeTests.cs new file mode 100644 index 0000000..0d16ab8 --- /dev/null +++ b/SemiPlot/SemiPlot.Tests.Unit/FollowEdgeTests.cs @@ -0,0 +1,88 @@ +using AwesomeAssertions; + +using SemiPlot.Tools.ArchiveSeeder; + +using Xunit; + +namespace SemiPlot.Tests.Unit; + +[Trait("Component", "Core")] +[Trait("Area", "Data")] +[Trait("Category", "Unit")] +public sealed class FollowEdgeTests +{ + private static readonly DateTime _seed = new(2026, 1, 2, 8, 0, 0, DateTimeKind.Unspecified); + + private static readonly TimeSpan _lag = TimeSpan.FromSeconds(60); + + [Fact] + public void AnEndPastTheEdgeReturnsTheWindowFromTheEdge() + { + var edge = new FollowEdge(_seed); + + var window = edge.Advance(_seed.AddSeconds(1)); + + window.Should().Be((_seed, _seed.AddSeconds(1))); + } + + [Theory] + [InlineData(0.0)] + [InlineData(-5.0)] + public void AnEndAtOrBehindTheEdgeReturnsNothingAndKeepsTheEdge(double secondsFromEdge) + { + var edge = new FollowEdge(_seed); + + var window = edge.Advance(_seed.AddSeconds(secondsFromEdge)); + + window.Should().BeNull(); + edge.Advance(_seed.AddSeconds(2)).Should().Be((_seed, _seed.AddSeconds(2))); + } + + // A one-second tick over a lagged clock that steps back twice, as a clock correction moves it. + [Fact] + public void ALaggedClockYieldsContiguousWindowsCoveringTheWholeSpan() + { + var wallClock = _seed + _lag; + double[] tickSeconds = [1.0, 1.0, -1.5, 2.0, 0.0, 1.0, -0.2, 3.0]; + var edge = new FollowEdge(_seed); + var windows = new List<(DateTime After, DateTime To)>(); + var nullCount = 0; + + foreach (var seconds in tickSeconds) + { + wallClock = wallClock.AddSeconds(seconds); + + if (edge.Advance(wallClock - _lag) is { } window) + { + windows.Add(window); + } + else + { + nullCount++; + } + } + + nullCount.Should().BeGreaterThan(0); + windows.Should().AllSatisfy(window => window.To.Should().BeAfter(window.After)); + windows[0].After.Should().Be(_seed); + windows.Zip(windows.Skip(1)).Should().AllSatisfy(pair => pair.Second.After.Should().Be(pair.First.To)); + windows[^1].To.Should().Be(wallClock - _lag); + } + + [Fact] + public void AFinalAdvanceAtTheUnlaggedClockCoversTheHeldBackSpan() + { + var wallClock = _seed + _lag; + var edge = new FollowEdge(_seed); + + for (var tick = 0; tick < 5; tick++) + { + wallClock = wallClock.AddSeconds(1); + edge.Advance(wallClock - _lag); + } + + var heldBack = edge.Advance(wallClock); + + heldBack.Should().Be((wallClock - _lag, wallClock)); + } +} diff --git a/SemiPlot/SemiPlot.Tests.Unit/LiveTailGeneratorTests.cs b/SemiPlot/SemiPlot.Tests.Unit/LiveTailGeneratorTests.cs index c04a374..0493f20 100644 --- a/SemiPlot/SemiPlot.Tests.Unit/LiveTailGeneratorTests.cs +++ b/SemiPlot/SemiPlot.Tests.Unit/LiveTailGeneratorTests.cs @@ -139,8 +139,53 @@ public void AChangeIntervalAtThePollIntervalEmitsNoAnchor() rows.Select(row => row.Timestamp).ToArray().Should().Equal(expected); } + [Fact] + public void ChangesAtALatticeInstantServesThatInstant() + { + var onTheLattice = _midnight.AddSeconds(9); + + var rows = LiveTailGenerator.ChangesAt(Options(), onTheLattice); + + rows.Should().AllSatisfy(row => row.Timestamp.Should().Be(onTheLattice)); + } + + [Fact] + public void ChangesAtBetweenInstantsServesTheEarlierOne() + { + var onTheLattice = _midnight.AddSeconds(9); + + var rows = LiveTailGenerator.ChangesAt(Options(), onTheLattice.AddMilliseconds(999.0)); + + rows.Should().AllSatisfy(row => row.Timestamp.Should().Be(onTheLattice)); + } + + [Fact] + public void ChangesAtServesOneRowPerPenInSelectionOrder() + { + var options = Options(); + + var rows = LiveTailGenerator.ChangesAt(options, _midnight.AddSeconds(9)); + + var selected = RawLayerGenerator.SelectPens(options.PenCount).Select(pen => pen.PenId); + + rows.Select(row => row.Id).Should().Equal(selected); + } + + [Theory] + [InlineData(1.0)] + [InlineData(0.05)] + public void ChangesAtServesNoAnchorRow(double changeSeconds) + { + var options = Options() with { PenCount = 1, ChangeSeconds = changeSeconds }; + var onTheLattice = _midnight.AddSeconds(9); + + var rows = LiveTailGenerator.ChangesAt(options, onTheLattice); + + rows.Should().ContainSingle().Which.Timestamp.Should().Be(onTheLattice); + } + private static FollowOptions Options() { - return new(BenchOptions.ConnectionString, TimeSpan.FromSeconds(1), 8, 1L, 1.0); + return new(BenchOptions.ConnectionString, TimeSpan.FromSeconds(1), 8, 1L, 1.0, TimeSpan.Zero, 55875); } } diff --git a/SemiPlot/SemiPlot.Tests.Unit/SeederCommandTests.cs b/SemiPlot/SemiPlot.Tests.Unit/SeederCommandTests.cs index d28a1e5..12b1869 100644 --- a/SemiPlot/SemiPlot.Tests.Unit/SeederCommandTests.cs +++ b/SemiPlot/SemiPlot.Tests.Unit/SeederCommandTests.cs @@ -65,7 +65,7 @@ public async Task ASeedingRunAcceptsEveryParameter() [Fact] public async Task AFollowRunAppliesDefaultsWhenOnlyRequiredOptionsAreGiven() { - var run = await RunAsync(["--connection", Connection, "--follow", "1"]); + var run = await RunAsync(["--connection", Connection, "--follow", "1", "--opc-port", "55875"]); run.ExitCode.Should().Be(0); run.Seed.Should().BeNull(); @@ -77,13 +77,18 @@ public async Task AFollowRunAppliesDefaultsWhenOnlyRequiredOptionsAreGiven() options.PenCount.Should().Be(SeederOptions.DefaultPenCount); options.Seed.Should().Be(SeederOptions.DefaultSeed); options.ChangeSeconds.Should().Be(SeederOptions.DefaultChangeSeconds); + options.ArchiveLag.Should().Be(TimeSpan.Zero); + options.OpcPort.Should().Be(55875); } [Fact] public async Task AFollowRunAcceptsEveryParameter() { var run = await RunAsync( - ["--connection", Connection, "--follow", "0.5", "--pens", "12", "--seed", "77", "--change-seconds", "2.5"]); + [ + "--connection", Connection, "--follow", "0.5", "--pens", "12", "--seed", "77", "--change-seconds", "2.5", + "--archive-lag", "120", "--opc-port", "4841" + ]); run.ExitCode.Should().Be(0); @@ -93,6 +98,49 @@ public async Task AFollowRunAcceptsEveryParameter() options.PenCount.Should().Be(12); options.Seed.Should().Be(77L); options.ChangeSeconds.Should().Be(2.5); + options.ArchiveLag.Should().Be(TimeSpan.FromSeconds(120)); + options.OpcPort.Should().Be(4841); + } + + [Fact] + public async Task AFollowRunAcceptsAnArchiveLagOfZero() + { + var run = await RunAsync( + ["--connection", Connection, "--follow", "1", "--opc-port", "55875", "--archive-lag", "0"]); + + run.ExitCode.Should().Be(0); + run.Follow.Should().BeOfType().Which.ArchiveLag.Should().Be(TimeSpan.Zero); + } + + [Fact] + public async Task AFollowRunWithoutAnOpcPortIsRejected() + { + var run = await RunAsync(["--connection", Connection, "--follow", "1"]); + + AssertRejectedWith(run, "--opc-port"); + AssertRejectedWith(run, "49875"); + } + + // Port 0 is refused because the OPC UA stack binds 4840 for it rather than a free port. + [Theory] + [InlineData("--archive-lag", "-1")] + [InlineData("--archive-lag", "NaN")] + [InlineData("--archive-lag", "86401")] + [InlineData("--archive-lag", "x")] + [InlineData("--opc-port", "0")] + [InlineData("--opc-port", "65536")] + [InlineData("--opc-port", "8.5")] + public async Task AFollowOnlyValueOutsideItsRangeIsRejectedNamingTheValue(string option, string value) + { + string[] required = option == "--opc-port" + ? ["--connection", Connection, "--follow", "1"] + : ["--connection", Connection, "--follow", "1", "--opc-port", "55875"]; + + var run = await RunAsync([.. required, option, value]); + + AssertRejectedWith(run, $"Option '{option}'"); + AssertRejectedWith(run, value); + run.Error.Should().NotContain("Unrecognized"); } // A follow run appends to an archive somebody else seeded. Each of these states a span or a tag @@ -103,11 +151,33 @@ public async Task AFollowRunAcceptsEveryParameter() [InlineData("--admin-connection", "Host=127.0.0.1;Database=archive;Username=postgres")] public async Task AFollowRunRejectsASeedingOption(string option, string value) { - var run = await RunAsync(["--connection", Connection, "--follow", "1", option, value]); + var run = await RunAsync(["--connection", Connection, "--follow", "1", "--opc-port", "55875", option, value]); AssertRejectedWith(run, option); } + [Theory] + [InlineData("--archive-lag", "120")] + [InlineData("--opc-port", "55875")] + public async Task ASeedingRunRejectsAFollowOption(string option, string value) + { + var run = await RunAsync(["--connection", Connection, "--end", EndText, option, value]); + + AssertRejectedWith(run, $"Option '{option}' belongs to a follow run"); + } + + [Fact] + public async Task AConvergeRunRejectsAnOpcPort() + { + var run = await RunAsync( + [ + "converge", "--connection", Connection, "--admin-connection", "Host=127.0.0.1;Database=postgres", + "--config-dir", "C:\\config", "--opc-port", "1" + ]); + + AssertRejectedWith(run, "--opc-port"); + } + [Theory] [InlineData("--days", "many")] [InlineData("--pens", "8.5")] @@ -186,8 +256,8 @@ public async Task ASeedingValueOutsideItsRangeIsRejected(string option, string v public async Task AFollowValueOutsideItsRangeIsRejected(string option, string value) { string[] required = option == "--follow" - ? ["--connection", Connection] - : ["--connection", Connection, "--follow", "1"]; + ? ["--connection", Connection, "--opc-port", "55875"] + : ["--connection", Connection, "--follow", "1", "--opc-port", "55875"]; AssertRejectedWith(await RunAsync([.. required, option, value]), option); } diff --git a/SemiPlot/SemiPlot.Tests.Unit/SharedLatticeTests.cs b/SemiPlot/SemiPlot.Tests.Unit/SharedLatticeTests.cs index e28f72f..f3ab539 100644 --- a/SemiPlot/SemiPlot.Tests.Unit/SharedLatticeTests.cs +++ b/SemiPlot/SemiPlot.Tests.Unit/SharedLatticeTests.cs @@ -30,7 +30,9 @@ public sealed class SharedLatticeTests TimeSpan.FromSeconds(1), PenCount, SeederOptions.DefaultSeed, - ChangeSeconds); + ChangeSeconds, + TimeSpan.Zero, + 55875); // A restart by hand, inside StaleArchiveGuard.MaximumAge of the edge the seeding run left. private static readonly DateTime _restartClock = BenchOptions.End.AddMinutes(3); @@ -89,4 +91,26 @@ public void AFollowRunResumingFromASeededEdgeRepeatsNoRowAndLeavesNoHole() seam.Should().BeGreaterThanOrEqualTo(TimeSpan.FromMilliseconds(1.0)).And.BeLessThanOrEqualTo(_changeInterval); } } + + // The third path: the OPC UA demo server serves ChangesAt. + [Fact] + public void TheServedChangeAtEveryLatticeInstantIsTheFollowWindowsChangeRow() + { + var options = _following with { ChangeSeconds = 0.5 }; + var interval = TimeSpan.FromSeconds(options.ChangeSeconds).Ticks; + var start = BenchOptions.End; + + var followed = LiveTailGenerator.Generate(options, start, start.AddHours(1)); + + var changes = followed + .Where(row => row.Timestamp.Ticks % interval == 0L) + .GroupBy(row => row.Timestamp) + .ToArray(); + + changes.Should().HaveCount(7200); + + var served = changes.SelectMany(change => LiveTailGenerator.ChangesAt(options, change.Key)); + + served.Should().Equal(changes.SelectMany(change => change)); + } } diff --git a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/DemoScadaNodeManager.cs b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/DemoScadaNodeManager.cs new file mode 100644 index 0000000..dcbfe24 --- /dev/null +++ b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/DemoScadaNodeManager.cs @@ -0,0 +1,157 @@ +using Opc.Ua; +using Opc.Ua.Server; + +using SemiPlot.DataSource.Postgres; + +namespace SemiPlot.Tools.ArchiveSeeder; + +/// The demo server's address space, shaped like Simple-Scada's own namespace. +internal sealed class DemoScadaNodeManager : CustomNodeManager2 +{ + private const string NamespaceUri = "http://simple-scada.com/UA"; + private const string DiagnosticsNamespaceUri = "http://opcfoundation.org/UA/Diagnostics"; + private const string ProjectId = "5E4D0C1A0B"; + private const string RootFolderName = "root"; + + // The real server gives _1 to its system folder, which the demo does not serve. + private const int RootFolderNumber = 2; + + private readonly FollowOptions _options; + private readonly ArchiveTimeConverter _converter; + private readonly DateTime _startUtc; + private readonly DateTime _startLocal; + private readonly Dictionary _variables = []; + + private DemoScadaNodeManager( + IServerInternal server, + ApplicationConfiguration configuration, + FollowOptions options, + ArchiveTimeConverter converter, + DateTime startUtc, + DateTime startLocal) + : base(server, configuration, NamespaceUri) + { + _options = options; + _converter = converter; + _startUtc = startUtc; + _startLocal = startLocal; + } + + public override void CreateAddressSpace(IDictionary> externalReferences) + { + lock (Lock) + { + if (!externalReferences.TryGetValue(ObjectIds.ObjectsFolder, out var objects)) + { + objects = []; + externalReferences[ObjectIds.ObjectsFolder] = objects; + } + + var pens = RawLayerGenerator.SelectPens(_options.PenCount); + var startValues = LiveTailGenerator.ChangesAt(_options, _startLocal) + .ToDictionary(row => row.Id, row => row.Value); + var root = CreateFolder(objects, RootFolderName, RootFolderNumber); + + foreach (var pen in pens) + { + var variable = CreateVariable(root, pen, startValues[pen.PenId]); + root.AddChild(variable); + _variables.Add(pen.PenId, variable); + } + + var groups = pens + .Where(pen => pen.PrimaryGroup.Length > 0) + .GroupBy(pen => pen.PrimaryGroup, StringComparer.Ordinal); + var folderNumber = RootFolderNumber; + + foreach (var group in groups) + { + var folder = CreateFolder(objects, group.Key, ++folderNumber); + + foreach (var pen in group) + { + folder.AddReference(ReferenceTypeIds.Organizes, false, _variables[pen.PenId].NodeId); + } + + AddPredefinedNode(SystemContext, folder); + } + + AddPredefinedNode(SystemContext, root); + } + } + + /// Writes each row into its pen's variable, stamped with the row's instant in UTC. + public void Write(IReadOnlyList rows) + { + lock (Lock) + { + foreach (var row in rows) + { + var variable = _variables[row.Id]; + variable.Value = new Variant(row.Value); + variable.Timestamp = _converter.ToUtc(row.Timestamp); + variable.ClearChangeMasks(SystemContext, false); + } + } + } + + private FolderState CreateFolder(IList objects, string name, int number) + { + var folder = new FolderState(null) + { + SymbolicName = name, + NodeId = new NodeId($"{ProjectId}{name}_{number}", NamespaceIndex), + BrowseName = new QualifiedName(name, NamespaceIndex), + DisplayName = new LocalizedText(name), + TypeDefinitionId = ObjectTypeIds.FolderType, + ReferenceTypeId = ReferenceTypeIds.Organizes, + }; + folder.AddReference(ReferenceTypeIds.Organizes, true, ObjectIds.ObjectsFolder); + objects.Add(new NodeStateReference(ReferenceTypeIds.Organizes, false, folder.NodeId)); + + return folder; + } + + private BaseDataVariableState CreateVariable(FolderState root, SyntheticPen pen, double value) + { + return new BaseDataVariableState(root) + { + SymbolicName = pen.Name, + NodeId = new NodeId($"{ProjectId}{pen.PenId}", NamespaceIndex), + BrowseName = new QualifiedName(pen.Name, NamespaceIndex), + DisplayName = new LocalizedText(pen.Name), + TypeDefinitionId = VariableTypeIds.BaseDataVariableType, + ReferenceTypeId = ReferenceTypeIds.Organizes, + DataType = DataTypeIds.Double, + ValueRank = ValueRanks.Scalar, + AccessLevel = AccessLevels.CurrentRead, + UserAccessLevel = AccessLevels.CurrentRead, + Value = new Variant(value), + StatusCode = StatusCodes.Good, + Timestamp = _startUtc, + }; + } + + /// Builds the manager during the server's start and keeps it for the publishing loop. + internal sealed class Factory( + FollowOptions options, + ArchiveTimeConverter converter, + DateTime startUtc, + DateTime startLocal) + : INodeManagerFactory + { + public DemoScadaNodeManager? Manager { get; private set; } + + public ArrayOf NamespacesUris => [NamespaceUri]; + + // Simple-Scada serves its variables at namespace index 3: registering Diagnostics first moves it from + // 3 to 2 and leaves 3 to the manager. + public INodeManager Create(IServerInternal server, ApplicationConfiguration configuration) + { + server.NamespaceUris.GetIndexOrAppend(DiagnosticsNamespaceUri); + Manager = new DemoScadaNodeManager(server, configuration, options, converter, startUtc, startLocal); + + return Manager; + } + } +} diff --git a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/DemoScadaServer.cs b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/DemoScadaServer.cs new file mode 100644 index 0000000..393757b --- /dev/null +++ b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/DemoScadaServer.cs @@ -0,0 +1,249 @@ +using System.Runtime.ExceptionServices; + +using Microsoft.Extensions.Logging; + +using Opc.Ua; +using Opc.Ua.Configuration; +using Opc.Ua.Server; + +using SemiPlot.DataSource.Postgres; + +namespace SemiPlot.Tools.ArchiveSeeder; + +/// +/// A Simple-Scada-shaped OPC UA server on loopback that serves the lattice the demo writer archives. +/// +public sealed class DemoScadaServer : IAsyncDisposable +{ + private const string ApplicationName = "SemiPlotDemoScada"; + + private readonly StandardServer _server; + private readonly CancellationTokenSource _stopping; + private int _disposed; + + private DemoScadaServer( + StandardServer server, + CancellationTokenSource stopping, + string endpointUrl, + Task completion) + { + _server = server; + _stopping = stopping; + EndpointUrl = endpointUrl; + Completion = completion; + } + + public string EndpointUrl { get; } + + /// + /// The publishing loop: it completes successfully only when cancels it; a fault + /// in the loop faults it. + /// + public Task Completion { get; } + + public static async Task StartAsync( + FollowOptions options, + TimeZoneInfo archiveZone, + string certificateDirectory, + ILoggerFactory loggerFactory) + { + var telemetry = new Telemetry(loggerFactory); + var endpointUrl = $"opc.tcp://127.0.0.1:{options.OpcPort}"; + var configuration = CreateConfiguration(endpointUrl, certificateDirectory, telemetry); + + await configuration.ValidateAsync(ApplicationType.Server, CancellationToken.None); + var certificateUsable = await new ApplicationInstance(configuration, telemetry) + .CheckApplicationInstanceCertificatesAsync(true, null, CancellationToken.None); + + if (!certificateUsable) + { + throw new SeederException( + $"The OPC UA application certificate under {certificateDirectory} is not usable; delete the " + + "directory and start again."); + } + + var converter = new ArchiveTimeConverter(archiveZone); + var startUtc = DateTime.UtcNow; + var startLocal = converter.ToArchiveLocal(startUtc); + var factory = new DemoScadaNodeManager.Factory(options, converter, startUtc, startLocal); + var server = new StandardServer(telemetry); + server.AddNodeManager(factory); + DemoScadaNodeManager manager; + + try + { + await server.StartAsync(configuration, CancellationToken.None); + manager = factory.Manager + ?? throw new InvalidOperationException("The OPC UA server started without its node manager."); + } + catch (ServiceResultException exception) when (exception.StatusCode == StatusCodes.BadNoCommunication) + { + await server.DisposeAsync(); + + throw new SeederException( + $"OPC UA port {options.OpcPort} is in use on 127.0.0.1; a running Simple-Scada holds " + + $"{FollowOptions.SimpleScadaPort} by default. Pass a free --opc-port.", + exception); + } + catch + { + await server.DisposeAsync(); + + throw; + } + + var stopping = new CancellationTokenSource(); + var completion = PublishAsync(manager, options, converter, startLocal, stopping.Token); + + return new DemoScadaServer(server, stopping, endpointUrl, completion); + } + + /// Stops the loop, stops the server and disposes it; a second call does nothing. + public async ValueTask DisposeAsync() + { + if (Interlocked.Exchange(ref _disposed, 1) == 1) + { + return; + } + + var faults = new List(); + + try + { + await _stopping.CancelAsync(); + await Completion; + } + catch (Exception exception) + { + faults.Add(exception); + } + + try + { + await _server.StopAsync(CancellationToken.None); + } + catch (Exception exception) + { + faults.Add(exception); + } + + try + { + await _server.DisposeAsync(); + } + catch (Exception exception) + { + faults.Add(exception); + } + + _stopping.Dispose(); + + if (faults.Count == 1) + { + ExceptionDispatchInfo.Throw(faults[0]); + } + + if (faults.Count > 1) + { + throw new AggregateException(faults); + } + } + + private static ApplicationConfiguration CreateConfiguration( + string endpointUrl, + string certificateDirectory, + ITelemetryContext telemetry) + { + return new ApplicationConfiguration(telemetry) + { + ApplicationName = ApplicationName, + ApplicationUri = $"urn:localhost:{ApplicationName}", + ApplicationType = ApplicationType.Server, + SecurityConfiguration = new SecurityConfiguration + { + ApplicationCertificate = new CertificateIdentifier + { + StoreType = CertificateStoreType.Directory, + StorePath = Path.Combine(certificateDirectory, "own"), + SubjectName = $"CN={ApplicationName}", + }, + TrustedIssuerCertificates = new CertificateTrustList + { + StoreType = CertificateStoreType.Directory, + StorePath = Path.Combine(certificateDirectory, "issuer"), + }, + TrustedPeerCertificates = new CertificateTrustList + { + StoreType = CertificateStoreType.Directory, + StorePath = Path.Combine(certificateDirectory, "trusted"), + }, + RejectedCertificateStore = new CertificateStoreIdentifier + { + StoreType = CertificateStoreType.Directory, + StorePath = Path.Combine(certificateDirectory, "rejected"), + }, + }, + TransportQuotas = new TransportQuotas(), + ServerConfiguration = new ServerConfiguration + { + BaseAddresses = [endpointUrl], + SecurityPolicies = + [ + new ServerSecurityPolicy + { + SecurityMode = MessageSecurityMode.None, + SecurityPolicyUri = SecurityPolicies.None, + }, + new ServerSecurityPolicy + { + SecurityMode = MessageSecurityMode.Sign, + SecurityPolicyUri = SecurityPolicies.Basic256Sha256, + }, + new ServerSecurityPolicy + { + SecurityMode = MessageSecurityMode.SignAndEncrypt, + SecurityPolicyUri = SecurityPolicies.Basic256Sha256, + }, + ], + UserTokenPolicies = [new UserTokenPolicy(UserTokenType.Anonymous)], + MaxRegistrationInterval = 0, + }, + }; + } + + /// Writes the newest lattice change, once per instant, until the token is cancelled. + private static async Task PublishAsync( + DemoScadaNodeManager manager, + FollowOptions options, + ArchiveTimeConverter converter, + DateTime startLocal, + CancellationToken cancellationToken) + { + var intervalTicks = RawLayerGenerator.ChangeIntervalTicks(options.ChangeSeconds); + var servedTicks = RawLayerGenerator.LatestChangeTicks(startLocal.Ticks, intervalTicks); + + try + { + while (true) + { + var now = converter.ToArchiveLocal(DateTime.UtcNow); + var changeTicks = RawLayerGenerator.LatestChangeTicks(now.Ticks, intervalTicks); + + if (changeTicks > servedTicks) + { + manager.Write(LiveTailGenerator.ChangesAt(options, now)); + servedTicks = changeTicks; + } + + var untilNextChange = TimeSpan.FromTicks(changeTicks + intervalTicks - now.Ticks); + var wholeMilliseconds = Math.Ceiling(untilNextChange.TotalMilliseconds); + await Task.Delay(TimeSpan.FromMilliseconds(wholeMilliseconds), cancellationToken); + } + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + } + } + + private sealed class Telemetry(ILoggerFactory loggerFactory) : TelemetryContextBase(loggerFactory); +} diff --git a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/FollowEdge.cs b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/FollowEdge.cs new file mode 100644 index 0000000..ab706a5 --- /dev/null +++ b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/FollowEdge.cs @@ -0,0 +1,24 @@ +namespace SemiPlot.Tools.ArchiveSeeder; + +/// +/// The archive's edge in a follow run: each window starts where the previous one ended, and the edge never +/// moves back, because a window behind it would regenerate rows the archive already holds. +/// +public sealed class FollowEdge(DateTime edge) +{ + private DateTime _edge = edge; + + /// The window (edge, end], advancing the edge; null, edge kept, when end is not past it. + public (DateTime After, DateTime To)? Advance(DateTime end) + { + if (end <= _edge) + { + return null; + } + + var window = (_edge, end); + _edge = end; + + return window; + } +} diff --git a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/FollowOptions.cs b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/FollowOptions.cs index e9c48b5..90299c2 100644 --- a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/FollowOptions.cs +++ b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/FollowOptions.cs @@ -6,9 +6,14 @@ public sealed record FollowOptions( TimeSpan Interval, int PenCount, long Seed, - double ChangeSeconds) + double ChangeSeconds, + TimeSpan ArchiveLag, + int OpcPort) { - // The ceiling on both --follow and --change-seconds: an interval above one day emits nothing and far - // above it overflows the tick arithmetic. + // The ceiling on --follow, --change-seconds and --archive-lag: an interval above one day emits nothing + // and far above it overflows the tick arithmetic. public const double MaximumSeconds = 86400.0; + + /// The port a running Simple-Scada holds by default. + internal const int SimpleScadaPort = 49875; } diff --git a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/LiveTailGenerator.cs b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/LiveTailGenerator.cs index 1346af3..95fdfd0 100644 --- a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/LiveTailGenerator.cs +++ b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/LiveTailGenerator.cs @@ -30,4 +30,14 @@ public static IReadOnlyList Generate(FollowOptions options, DateTime return rows; } + + // The change rows at the newest lattice instant at or before `instant`, one per pen. A window one tick + // wide holds the change row and never its anchor, which lies one poll interval earlier. + public static IReadOnlyList ChangesAt(FollowOptions options, DateTime instant) + { + var interval = RawLayerGenerator.ChangeIntervalTicks(options.ChangeSeconds); + var change = new DateTime(RawLayerGenerator.LatestChangeTicks(instant.Ticks, interval)); + + return Generate(options, change.AddTicks(-1), change); + } } diff --git a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/Program.cs b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/Program.cs index 0a3f1b2..d0d35dc 100644 --- a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/Program.cs +++ b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/Program.cs @@ -1,9 +1,14 @@ +using Microsoft.Extensions.Logging; + using Npgsql; namespace SemiPlot.Tools.ArchiveSeeder; public static class Program { + private static readonly string _opcCertificateDirectory = + Path.Combine(Path.GetTempPath(), "SemiPlot", "OpcDemoServer"); + public static Task Main(string[] arguments) { return SeederCommand.RunAsync(arguments, options => ReportingAsync(() => SeedAsync(options)), @@ -42,39 +47,40 @@ private static async Task SeedAsync(SeederOptions options) return 0; } - // The demo writer: each tick appends the span since the previous tick and thins it into the coarse layers. + // docs/architecture/bench.md#the-demo-writer private static async Task FollowAsync(FollowOptions options) { + // Disposed at method exit, after the server's explicit dispose, so its queue flushes the server's last lines. + using var loggerFactory = CreateServerLoggerFactory(); var stopping = new CancellationTokenSource(); // Ctrl+C stops the loop where it waits rather than tearing the process down inside a COPY: the // in-flight append is never handed the token. Console.CancelKeyPress += StopOnCancel; - ReportFollowPlan(options); - // Read once, before anything is written: the edge the first tick continues from. - var newest = await StaleArchiveGuard.CheckAsync(options.ConnectionString, LocalNow()); + var newest = await StaleArchiveGuard.CheckAsync(options.ConnectionString, ArchiveClock(options)); + // Open at the edge: the edge row already sits on the lattice this run walks again. + var edge = new FollowEdge(newest ?? ArchiveClock(options)); var writer = new ArchiveWriter(options.ConnectionString); - // Open at the edge: the edge row already sits on the lattice this run walks again. - var edge = newest ?? LocalNow(); + var server = await DemoScadaServer.StartAsync( + options, TimeZoneInfo.Local, _opcCertificateDirectory, loggerFactory); - while (await WaitForTickAsync(options.Interval, stopping.Token)) + try { - var now = LocalNow(); - var appended = await AppendAsync(writer, options, edge, now); - - // After the append has committed, never before it: at a cadence longer than a period the tick's - // own rows are the closing period's last ones, and a flush ahead of them would pick a coarse - // 'last' row the period had not produced yet. - var thinned = await CoarseFlush.FlushAsync(options, edge, now); + ReportFollowPlan(options, server); + await AppendUntilStoppedAsync(writer, options, edge, server, stopping.Token); + } + catch (Exception primary) + { + await DisposeBesideAsync(server, primary, loggerFactory.CreateLogger()); - Console.WriteLine($"appended {appended} rows, {thinned} coarse, up to {now:O}"); - edge = now; + throw; } + await server.DisposeAsync(); Console.CancelKeyPress -= StopOnCancel; stopping.Dispose(); Console.WriteLine("stopped"); @@ -88,6 +94,46 @@ void StopOnCancel(object? sender, ConsoleCancelEventArgs pressed) } } + private static async Task AppendUntilStoppedAsync( + ArchiveWriter writer, + FollowOptions options, + FollowEdge edge, + DemoScadaServer server, + CancellationToken cancellationToken) + { + while (await WaitForTickAsync(options.Interval, cancellationToken)) + { + await ThrowIfServerEndedAsync(server.Completion); + + if (edge.Advance(ArchiveClock(options)) is { } window) + { + await AppendAndFlushAsync(writer, options, window.After, window.To); + } + } + + // A graceful stop writes the span the lag held back. + if (edge.Advance(LocalNow()) is { } heldBack) + { + await AppendAndFlushAsync(writer, options, heldBack.After, heldBack.To); + } + } + + private static async Task DisposeBesideAsync(DemoScadaServer server, Exception primary, ILogger logger) + { + try + { + await server.DisposeAsync(); + } + catch (Exception disposal) + { + // The dispose rethrows a publishing-loop fault as the same object the tick loop already threw. + if (!ReferenceEquals(disposal, primary)) + { + logger.LogError(disposal, "The OPC UA demo server failed to stop"); + } + } + } + // The archive column is 'timestamp(3) without time zone' holding the SCADA host's naive local time // (docs/architecture/scada-archive.md#time-semantics). private static DateTime LocalNow() @@ -95,6 +141,48 @@ private static DateTime LocalNow() return DateTime.SpecifyKind(DateTime.Now, DateTimeKind.Unspecified); } + private static DateTime ArchiveClock(FollowOptions options) + { + return LocalNow() - options.ArchiveLag; + } + + // No configuration source, so the console-formatter variables the AppHost injects stay inert. + private static ILoggerFactory CreateServerLoggerFactory() + { + return LoggerFactory.Create(builder => builder + .SetMinimumLevel(LogLevel.Warning) + .AddSimpleConsole(console => console.SingleLine = true) + .AddConsole(console => console.LogToStandardErrorThreshold = LogLevel.Warning)); + } + + private static async Task ThrowIfServerEndedAsync(Task completion) + { + if (!completion.IsCompleted) + { + return; + } + + await completion; + + throw new InvalidOperationException("The OPC UA publishing loop ended."); + } + + private static async Task AppendAndFlushAsync( + ArchiveWriter writer, + FollowOptions options, + DateTime after, + DateTime to) + { + var appended = await AppendAsync(writer, options, after, to); + + // After the append has committed, never before it: at a cadence longer than a period the tick's own + // rows are the closing period's last ones, and a flush ahead of them would pick a coarse 'last' row + // the period had not produced yet. + var thinned = await CoarseFlush.FlushAsync(options, after, to); + + Console.WriteLine($"appended {appended} rows, {thinned} coarse, up to {to:O}"); + } + private static async Task WaitForTickAsync(TimeSpan interval, CancellationToken cancellationToken) { try @@ -119,12 +207,14 @@ private static async Task AppendAsync(ArchiveWriter writer, FollowOptions : await writer.WriteAsync(rows, after, to.AddTicks(1), allowExistingRows: true); } - private static void ReportFollowPlan(FollowOptions options) + private static void ReportFollowPlan(FollowOptions options, DemoScadaServer server) { Console.WriteLine($"mode follow, every {options.Interval.TotalSeconds:0.###} s"); Console.WriteLine($"pens {options.PenCount}"); Console.WriteLine($"seed {options.Seed}"); Console.WriteLine($"change seconds {options.ChangeSeconds}"); + Console.WriteLine($"archive lag {options.ArchiveLag.TotalSeconds:0.###} s"); + Console.WriteLine($"opc {server.EndpointUrl}, {options.PenCount} variables"); } private static async Task ConvergeAsync(ConvergeOptions options) diff --git a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/RawLayerGenerator.cs b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/RawLayerGenerator.cs index 4f73ab8..f2b3473 100644 --- a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/RawLayerGenerator.cs +++ b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/RawLayerGenerator.cs @@ -178,6 +178,12 @@ private static long FirstIndex(long fromTicks, long intervalTicks) return Math.Max(1L, (fromTicks + intervalTicks - 1) / intervalTicks); } + /// The newest change instant at or before , never below index 1. + internal static long LatestChangeTicks(long atTicks, long intervalTicks) + { + return Math.Max(1L, atTicks / intervalTicks) * intervalTicks; + } + private static double ValueAt(long seed, SyntheticPen pen, long index) { return pen.LogScaleOnStart diff --git a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/SeederCommand.cs b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/SeederCommand.cs index 55fa4a7..2b4c0e5 100644 --- a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/SeederCommand.cs +++ b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/SeederCommand.cs @@ -2,6 +2,7 @@ using System.CommandLine.Invocation; using System.CommandLine.Parsing; using System.Globalization; +using System.Net; namespace SemiPlot.Tools.ArchiveSeeder; @@ -56,7 +57,8 @@ public static class SeederCommand { Description = "Interval between value changes, in seconds.", DefaultValueFactory = _ => SeederOptions.DefaultChangeSeconds, - CustomParser = result => ParseSeconds(result, "--change-seconds") ?? SeederOptions.DefaultChangeSeconds + CustomParser = result => + ParseSeconds(result, "--change-seconds", acceptsZero: false) ?? SeederOptions.DefaultChangeSeconds }; private static readonly Option _follow = new("--follow") @@ -64,7 +66,22 @@ public static class SeederCommand Description = "Seconds between ticks, and the switch that selects the demo writer: each tick appends " + "the raw rows of the wall-clock span since the previous one and thins them into the coarse " + "layers. It seeds nothing.", - CustomParser = result => ParseSeconds(result, "--follow") + CustomParser = result => ParseSeconds(result, "--follow", acceptsZero: false) + }; + + private static readonly Option _archiveLag = new("--archive-lag") + { + Description = "Seconds the archive inserts trail the clock, the way a SCADA's buffered archive trails the " + + "plant. The OPC UA server is not held back. Follow only.", + DefaultValueFactory = _ => 0.0, + CustomParser = result => ParseSeconds(result, "--archive-lag", acceptsZero: true) ?? 0.0 + }; + + private static readonly Option _opcPort = new("--opc-port") + { + Description = "Port the OPC UA demo server listens on, on 127.0.0.1. Required for a follow run; a running " + + $"Simple-Scada holds {FollowOptions.SimpleScadaPort} by default.", + CustomParser = result => ParseWholeNumber(result, "--opc-port", 1, IPEndPoint.MaxPort) }; private static readonly (Option Option, string Name)[] _seedingOnly = @@ -74,6 +91,12 @@ private static readonly (Option Option, string Name)[] _seedingOnly = (_adminConnection, "--admin-connection") ]; + private static readonly (Option Option, string Name)[] _followOnly = + [ + (_archiveLag, "--archive-lag"), + (_opcPort, "--opc-port") + ]; + private static readonly Option _convergeAdminConnection = new("--admin-connection") { Description = "Npgsql connection string on the maintenance database, for the superuser role.", @@ -124,7 +147,8 @@ private static RootCommand Build() foreach (var option in new Option[] { - _connection, _end, _adminConnection, _days, _pens, _seed, _changeSeconds, _follow + _connection, _end, _adminConnection, _days, _pens, _seed, _changeSeconds, _follow, _archiveLag, + _opcPort }) { root.Options.Add(option); @@ -213,8 +237,8 @@ private static long ParseSeed(ArgumentResult result) } // NaN fails every comparison, so a bare `<= 0` check lets it through, and Infinity survives it too and - // overflows the interval arithmetic. - private static double? ParseSeconds(ArgumentResult result, string name) + // overflows the interval arithmetic. The refusal quotes the token, which no culture renders differently. + private static double? ParseSeconds(ArgumentResult result, string name, bool acceptsZero) { var text = result.Tokens[^1].Value; @@ -225,11 +249,15 @@ private static long ParseSeed(ArgumentResult result) return null; } - if (!double.IsFinite(seconds) || seconds <= 0.0 || seconds > FollowOptions.MaximumSeconds) + var belowFloor = acceptsZero ? seconds < 0.0 : seconds <= 0.0; + + if (!double.IsFinite(seconds) || belowFloor || seconds > FollowOptions.MaximumSeconds) { + var floor = acceptsZero ? "at least 0" : "greater than 0"; + result.AddError( - $"Option '{name}' must be a finite number greater than 0 and at most " - + $"{FollowOptions.MaximumSeconds:0}, got {seconds}."); + $"Option '{name}' must be a finite number {floor} and at most " + + $"{FollowOptions.MaximumSeconds:0}, got '{text}'."); return null; } @@ -300,28 +328,46 @@ private static Task Interpret( return Report(errors); } - var connection = parsed.GetValue(_connection)!; - var pens = parsed.GetValue(_pens); - var seedValue = parsed.GetValue(_seed); - var changeSeconds = parsed.GetValue(_changeSeconds); + return parsed.GetValue(_follow) is { } seconds + ? InterpretFollow(parsed, seconds, follow) + : InterpretSeeding(parsed, seed); + } + + private static Task InterpretFollow(ParseResult parsed, double seconds, Func> follow) + { + var errors = ExplicitlyGiven(parsed, _seedingOnly) + .Select(name => $"Option '{name}' belongs to a seeding run: --follow appends to an archive somebody " + + "else seeded, so it states no span and fills no tag catalogue.") + .ToList(); - if (parsed.GetValue(_follow) is { } seconds) + if (parsed.GetValue(_opcPort) is not { } opcPort) { - foreach (var (option, name) in _seedingOnly) - { - if (parsed.GetResult(option) is { Implicit: false }) - { - errors.Add( - $"Option '{name}' belongs to a seeding run: --follow appends to an archive somebody else " - + "seeded, so it states no span and fills no tag catalogue."); - } - } + errors.Add( + "Option '--opc-port' is required for a follow run: a running Simple-Scada holds " + + $"{FollowOptions.SimpleScadaPort} by default, so the demo server takes no port of its own choosing."); - return errors.Count > 0 - ? Report(errors) - : follow(new FollowOptions(connection, TimeSpan.FromSeconds(seconds), pens, seedValue, changeSeconds)); + return Report(errors); } + return errors.Count > 0 + ? Report(errors) + : follow(new FollowOptions( + parsed.GetValue(_connection)!, + TimeSpan.FromSeconds(seconds), + parsed.GetValue(_pens), + parsed.GetValue(_seed), + parsed.GetValue(_changeSeconds), + TimeSpan.FromSeconds(parsed.GetValue(_archiveLag)), + opcPort)); + } + + private static Task InterpretSeeding(ParseResult parsed, Func> seed) + { + var errors = ExplicitlyGiven(parsed, _followOnly) + .Select(name => $"Option '{name}' belongs to a follow run: a seeding run writes a closed span and " + + "serves nothing live.") + .ToList(); + if (parsed.GetValue(_end) is not { } end) { errors.Add("Option '--end' is required: a floating 'now' would make two runs of the same seed differ."); @@ -331,6 +377,7 @@ private static Task Interpret( var days = parsed.GetValue(_days); var span = TimeSpan.FromDays(days); + var changeSeconds = parsed.GetValue(_changeSeconds); if (changeSeconds > span.TotalSeconds) { @@ -341,16 +388,23 @@ private static Task Interpret( return errors.Count > 0 ? Report(errors) : seed(new SeederOptions( - connection, + parsed.GetValue(_connection)!, end, days, - pens, - seedValue, + parsed.GetValue(_pens), + parsed.GetValue(_seed), changeSeconds, SeederOptions.DefaultBreakCount, parsed.GetValue(_adminConnection))); } + private static IEnumerable ExplicitlyGiven(ParseResult parsed, (Option Option, string Name)[] options) + { + return options + .Where(entry => parsed.GetResult(entry.Option) is { Implicit: false }) + .Select(entry => entry.Name); + } + private static Task Report(IReadOnlyList errors) { foreach (var error in errors) diff --git a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/SeederException.cs b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/SeederException.cs index 26df607..1f0afb5 100644 --- a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/SeederException.cs +++ b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/SeederException.cs @@ -4,4 +4,5 @@ namespace SemiPlot.Tools.ArchiveSeeder; /// A refusal the operator can act on: the archive is not what the run needs. The entry point prints the /// message and exits 1; anything else that escapes is a fault in this tool and keeps its stack trace. /// -public sealed class SeederException(string message) : Exception(message); +public sealed class SeederException(string message, Exception? innerException = null) + : Exception(message, innerException); diff --git a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/SemiPlot.Tools.ArchiveSeeder.csproj b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/SemiPlot.Tools.ArchiveSeeder.csproj index be7820a..b26bffe 100644 --- a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/SemiPlot.Tools.ArchiveSeeder.csproj +++ b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/SemiPlot.Tools.ArchiveSeeder.csproj @@ -5,12 +5,15 @@ + + + diff --git a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/StaleArchiveGuard.cs b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/StaleArchiveGuard.cs index c40007b..4c3e00b 100644 --- a/SemiPlot/SemiPlot.Tools.ArchiveSeeder/StaleArchiveGuard.cs +++ b/SemiPlot/SemiPlot.Tools.ArchiveSeeder/StaleArchiveGuard.cs @@ -4,7 +4,6 @@ namespace SemiPlot.Tools.ArchiveSeeder; -// Bounds the first tick of Program.FollowAsync: an archive further behind the clock than MaximumAge is refused. // docs/architecture/bench.md#the-demo-writer public static class StaleArchiveGuard { @@ -47,9 +46,10 @@ private static string Describe(DateTime newest, TimeSpan age) { var behind = age.TotalMinutes.ToString("0.#", CultureInfo.InvariantCulture); - return $"public.trends ends at {newest:O}, {behind} minutes behind this machine's clock, and a " - + "follow run continues from that edge: the first tick would write the whole span at once, " - + "and a day partition for each day it covers. Refill the archive up to now with " - + "`converge`, or point --connection at an archive a writer is already keeping live."; + return $"public.trends ends at {newest:O}, {behind} minutes behind this machine's clock less " + + "--archive-lag, and a follow run continues from that edge: the first tick would write the " + + "whole span at once, and a day partition for each day it covers. Refill the archive up to now " + + "with `converge`, pass the --archive-lag the previous run used, or point --connection at an " + + "archive a writer is already keeping live."; } } diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 6974ccc..1ea18e6 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -39,9 +39,9 @@ implementation plans see `docs/plans/`. - [sources.md](./sources.md) — citation convention and the registry every factual claim resolves against: vendor manual pages, vendor forum topics, vendor support answers, our own measurements, our own decisions. -- [bench.md](./bench.md) — the seeded PostgreSQL bench: the seeder and the demo writer, the lattice - they share, the layer thinning, the container fixture and its template-and-clone lifecycle, the - application bench recipe, and the headless render and input guards. +- [bench.md](./bench.md) — the seeded PostgreSQL bench: the seeder and the demo writer with its OPC UA + demo server, the lattice they share, the layer thinning, the container fixture and its + template-and-clone lifecycle, the application bench recipe, and the headless render and input guards. - [testing-strategy.md](./testing-strategy.md) — what each kind of test owns: the unit / integration / end-to-end distinction drawn on this repository's own files, why the two test diff --git a/docs/architecture/bench.md b/docs/architecture/bench.md index 5443015..33346dc 100644 --- a/docs/architecture/bench.md +++ b/docs/architecture/bench.md @@ -12,6 +12,7 @@ is remains in `scada-archive.md`; what SemiPlot reads from it remains in `data-i | The daily partitions and the rows | `SemiPlot.Tools.ArchiveSeeder`, connected as `scada_writer` | | The catalogue rows in `semiplot_tags`, `semiplot_groups` and `semiplot_pen_groups` | `SemiPlot.Tools.ArchiveSeeder` over the admin connection, written once when the stand comes up | | The moving live edge of a demo archive | `SemiPlot.Tools.ArchiveSeeder --follow`, connected as `scada_writer` | +| The OPC UA demo server on `127.0.0.1`, the port `--opc-port` names | `SemiPlot.Tools.ArchiveSeeder --follow`, through `DemoScadaServer` (`#the-opc-ua-demo-server`) | | Template build, the clones, teardown | `SemiPlot.Tests.Integration` | The archive table is the provisioner's, not this repository's: nothing here transcribes the vendor's @@ -77,35 +78,59 @@ run. ## The demo writer `--follow ` runs the seeder as a demo writer: it appends to an archive somebody else seeded, -so it plants no break and fills no tag catalogue. `--end`, `--days` and `--admin-connection` are -rejected in this mode. `--pens`, `--seed` and `--change-seconds` mean what they mean in a seeding run. -`--follow` and `--change-seconds` each take a finite number of seconds above 0 and at most one day, -`FollowOptions.MaximumSeconds` (86 400), in every run. +so it plants no break and fills no tag catalogue, and it serves the lattice over OPC UA +(`#the-opc-ua-demo-server`). `--end`, `--days` and `--admin-connection` are rejected in this mode. +`--archive-lag` and `--opc-port` belong to it alone: a seeding run that names either fails naming it, +and `converge` does not know them. `--pens`, `--seed` and `--change-seconds` mean what they mean in a +seeding run. `--follow` and `--change-seconds` each take a finite number of seconds above 0 and at most +one day, `FollowOptions.MaximumSeconds` (86 400), in every run. `--archive-lag` takes the same range +with 0 included, and 0 is its default. ```powershell dotnet run --project SemiPlot/SemiPlot.Tools.ArchiveSeeder/SemiPlot.Tools.ArchiveSeeder.csproj -- ` --connection "Host=127.0.0.1;Port=55432;Database=semiplot_app;Username=scada_writer;Password=" ` - --follow 1 --change-seconds 0.5 + --follow 1 --change-seconds 0.5 --opc-port 55875 ``` -Every tick appends the raw rows of the window since the previous tick, thins them into the coarse -layers and prints both counts; `Ctrl+C` stops the loop where it waits, never inside an append. +The run reads the archive's edge, starts the OPC UA server, and only then prints its plan, which ends +with the `archive lag` line and the `opc` line naming the endpoint and the variable count. Every tick +first checks the server's publishing loop: a loop that faulted rethrows its fault, and a loop that +ended without one is an `InvalidOperationException`, so the writer never appends beside a dead server. +The tick then appends the raw rows of the window the archive clock has passed since the previous tick, +thins them into the coarse layers and prints both counts. `Ctrl+C` stops the loop where it waits, +never inside an append; the run then writes the held-back span, stops the server, which takes about +5 s when a client session is attached, and prints `stopped`. A tick that fails stops the server too, +and the run exits on the tick's own failure: a server that also fails to stop only logs that second +failure. To see a live break on the stand, insert one raw row for a pen in view as `scada_writer`, with `l = 0`, `q = 32` and `t` past the newest row: the line ends at its value and resumes at the writer's next row (`data-integration.md#realtime`). -- **The window is `after < t <= to`.** `LiveTailGenerator.Generate(options, after, to)` takes the - archive's edge on the first tick and the previous tick's own instant after that. Both are instants - the archive already accounts for, so consecutive windows partition the lattice: a restart writes no - row twice — a `COPY` has no conflict handling, and the edge sits on a lattice point — and leaves - no hole, since the next lattice point is inside one change interval. `FollowRestartTests` performs - the restart against a database; `SharedLatticeTests` performs it in memory. +- **The window is `after < t <= to`.** `LiveTailGenerator.Generate(options, after, to)` runs over the + window `FollowEdge.Advance(archive clock)` returns. `FollowEdge` holds the archive's edge, seeded + with the archive's newest row: for an end past the edge it returns `(edge, end]` and moves the edge + to that end, and while the end has not passed the edge it returns nothing and keeps the edge. The + edge never moves back, because a window behind it would regenerate rows the archive already holds. + Every window opens at an instant the archive already accounts for, so consecutive windows partition the + lattice: a restart writes no row twice — a `COPY` has no conflict handling, and the edge sits on a + lattice point — and leaves no hole, since the next lattice point is inside one change interval. + `FollowRestartTests` performs the restart against a database; `SharedLatticeTests` performs it in + memory; `FollowEdgeTests` pins the windows. +- **`--archive-lag` holds the archive back.** The archive clock is the local clock less + `--archive-lag`, so the archive trails the OPC UA stream by the lag, the way the SCADA's buffered + archive trails the plant. The OPC UA server is not lagged. `CoarseFlush` runs over the same window + as the append, because its opening-row insert reads the raw rows just written. A graceful stop + writes the held-back span: after `Ctrl+C` the run advances the edge once to the unlagged clock and + appends and thins that window, as the SCADA flushes its buffer on a project stop. A killed writer + loses the span, as a killed SCADA does. - **`StaleArchiveGuard` bounds the first tick.** It reads `max(t)` once and refuses an archive more - than `MaximumAge` (five minutes) behind the clock, naming `converge` as the refill: - the first tick writes everything between the edge and now, and against an archive filled weeks ago - that would be weeks of rows and a partition per day. An empty archive is accepted and the loop - starts at the clock. + than `MaximumAge` (five minutes) behind the archive clock: the first tick writes everything between + the edge and that clock, and against an archive filled weeks ago that would be weeks of rows and a + partition per day. The guard reads the lagged clock, so a lag longer than five minutes passes it. A + restart that drops the lag the previous run used meets an archive behind the unlagged clock, and the + refusal names both ways out: refill with `converge`, or pass the previous run's `--archive-lag`. An + empty archive is accepted, and the edge starts at the archive clock. - **Every layer moves, each on its own cadence.** `CoarseFlush` works one layer at a time on a connection of its own: for every period the tick leaves behind it reads the finer layer's rows, runs `LayerThinner.Thin` over them and inserts the result, and an opening-row `INSERT` writes the @@ -116,7 +141,126 @@ To see a live break on the stand, insert one raw row for a pen in view as `scada start and the distance to the live edge peaks at exactly one period, which the non-strict comparison keeps. - **The clock is the machine's local wall clock with its `Kind` stripped**, because the archive - column holds the SCADA host's naive local time. + column holds the SCADA host's naive local time. The archive clock subtracts the lag from it. The + OPC UA server reads its own clock through `ArchiveTimeConverter` over `TimeZoneInfo.Local` + (`#the-opc-ua-demo-server`). + +### The OPC UA demo server + +`DemoScadaServer` in `SemiPlot.Tools.ArchiveSeeder` is a Simple-Scada-shaped OPC UA server on the OPC +Foundation stack 2.0.1, with `DemoScadaNodeManager` as its address space. The follow run starts it +after the guard read and before the first tick, over the pens it archives, +`RawLayerGenerator.SelectPens(--pens)`, and disposes it after the last write. Every value it serves is +a row `LiveTailGenerator` emits, so no second source of values exists. + +**The address space.** Namespace `http://simple-scada.com/UA` sits at index 3, where the real server +has it and where the vendor calls `ns=3;s=` stable. The manager's factory +registers `http://opcfoundation.org/UA/Diagnostics` first, which takes index 2 and leaves 3 to the +manager. The project ID is `5E4D0C1A0B`: ten hex characters, the shape of a real project ID, and +different from the one measured on the real server, so a client that hard-codes that ID fails here. Every browse name carries namespace index 3. +`Objects` organizes, in this order: + +| Node | Browse name | Organizes | +| --- | --- | --- | +| `ns=3;s=5E4D0C1A0Broot_2` | `root` | every served pen | +| `ns=3;s=5E4D0C1A0B_`, `n` from 3 | the group name | the pens whose `PrimaryGroup` it is | + +The group folders follow the first appearance of each non-empty `PrimaryGroup` in the selection. A +pen's variable is `ns=3;s=5E4D0C1A0B`, its browse and display name `SyntheticPen.Name`, of +`DataType` Double, scalar, current-read access, status `Good`; a group folder references the same +variable node `root` holds. The ungrouped pen, `UncommissionedPenId` 5001, sits under `root` only, and +a pen's later groups get no folder: `TwoGroupPenId` appears under `Heaters` and never under +`Watchlist`. The real server gives `_1` to its system-variables folder, which the demo does not serve. +A client finds `root` by browse name, never by position or by a fixed node id. + +**The start stamp.** `StartAsync` reads the UTC clock once, ahead of the stack's start. Every variable +starts at the value of the newest lattice change at or before that instant, stamped with the instant +itself, so a client's first notification after a start carries the start instant, as on the real +server after a project start. The stamp holds until the loop's first write, the next lattice change. +The stack takes 1.5 to 4.3 s to start, and a change that falls inside the start is written as soon as +the loop runs. A client therefore sees the start stamp only when it subscribes before the first change +after the start: at the stand's 0.5 s lattice it never does, and a test that restarts the server to +observe the start stamp needs a change interval longer than the start and the subscribe together. + +**The publishing rule.** One loop wakes at each lattice instant. It reads the clock as +`ArchiveTimeConverter.ToArchiveLocal(DateTime.UtcNow)`, takes the newest lattice instant at or before +it (`RawLayerGenerator.LatestChangeTicks`), and, when that instant is newer than the last one served, +writes `LiveTailGenerator.ChangesAt` there into the variables: the value, `SourceTimestamp = +ArchiveTimeConverter.ToUtc(row.Timestamp)`, then `ClearChangeMasks`. `ChangesAt` runs `Generate` over a +window one tick wide, so `AppendWindow` stays the one emitter and the server writes the change row and +never its pre-anchor: the anchor repeats the previous value one poll interval early, which is the +archive's shape, not a change the plant made. `SharedLatticeTests` pins the served change to the +follow window's change row at every instant of an hour. Each variable holds one value, so a stall that +skips instants publishes the newest one only; the archive still receives every row. A live value and +the archive row that later replaces it map to one UTC instant through the converter the viewer runs. +Outside the two DST transitions the stream and the archive's change rows agree row for row. At the +autumn fall-back the local clock repeats an hour: its first pass stamps resolve to standard time, an +hour later than true UTC, as the archive rows of that hour do in the viewer, and during the second pass +the newest instant stays behind the last one served, so the server publishes nothing new for that +hour. At the spring jump the archive tick writes lattice rows at local instants that never existed, +while the server publishes only the instant after the jump. + +**Endpoints and the anonymous token.** The server binds and advertises `opc.tcp://127.0.0.1:` +and listens on `127.0.0.1` only. It offers three endpoints: `None`, and `Sign` and `SignAndEncrypt` +under `Basic256Sha256`, one of the three secured policies the real server offers. No client +certificate is trusted, so only `None` admits a session; the secured endpoints exist so a client that +picks one fails here as it would on the real server. Each endpoint carries one anonymous token policy, +the stack's stock `UserTokenPolicy(Anonymous)`, with no override of any kind. The stock stack +advertises it with `SecurityPolicyUri = http://opcfoundation.org/UA/SecurityPolicy#None` on the `None` +endpoint only, and with an empty URI on `Sign` and `SignAndEncrypt`; the real server shows `#None` on +all seven of its endpoints (both measured 2026-10-20). A client takes the anonymous token policy from +the `None` endpoint's own `UserIdentityTokens`, which works against both servers; a client that builds +one with a null policy URI fails `BadSecurityChecksFailed` against the demo. +`MaxRegistrationInterval = 0` keeps the stack from registering with a discovery server at +`opc.tcp://localhost:4840`. + +**The certificate directory.** The stack runs no server without an application certificate, not even +one that offers only `None`. `StartAsync` keeps the certificate in the directory stores `own`, +`issuer`, `trusted` and `rejected` under the directory its caller names; +`CheckApplicationInstanceCertificatesAsync` creates a self-signed one on the first start and reuses it +after that, and a certificate it reports unusable is a `SeederException` that names the directory and +asks to delete it. The follow run passes the fixed `%TEMP%\SemiPlot\OpcDemoServer`, so every run, +stopped or killed, reuses one certificate. Nothing removes that directory: it holds the server's +certificate and private key, and `rejected/certs` collects the certificate of every client refused on +a secured endpoint. `DemoDirectories` sweeps only `ConfigFiles` and `Logs` beside it +(`#the-demos-directories`). Whether the check replaces an expired certificate is not verified. + +**The console log.** The stack logs through a `LoggerFactory` built with `LoggerFactory.Create` and no +configuration source: the console provider, `Warning` and above, one line per record, all of it on +standard error. A rejected session therefore leaves a line on the stand, and the console-formatter +variables the AppHost injects stay inert. The factory is disposed after the server, so the server's +last lines reach the console. One error line is expected on every session: `Opc.Ua.CertificateValidationCore` +logs "The domain 'opc.tcp://127.0.0.1:/' is not listed in the server certificate" at `Error`, not +`Warning`, so the console prints it, and nothing suppresses it (measured 2026-10-20). + +**The port.** `--opc-port` takes a whole number from 1 to 65535 and is required for a follow run, with +no default: the real Simple-Scada listens on 49875 on the operator's machine, and a writer that took a +default port would either fail beside it or take the port from it. A follow run without it fails with +"Option '--opc-port' is required for a follow run: a running Simple-Scada holds 49875 by default, so +the demo server takes no port of its own choosing." Port 0 is refused at the command line, because +the stack asked for port 0 binds 4840, the OPC UA default, rather than a free port. A busy port fails +the start, and the run exits 1 with one line, "OPC UA port is in use on 127.0.0.1; a running +Simple-Scada holds 49875 by default. Pass a free --opc-port.", a `SeederException` that carries the +stack's `BadNoCommunication` `ServiceResultException` as its inner exception. + +**The lifetime.** `DemoScadaServer.Completion` is the publishing loop: it completes successfully only +when `DisposeAsync` cancels it, and a fault in the loop faults it. `DisposeAsync` cancels the loop, stops the stack and +disposes it, attempts all three, rethrows only a real fault, aggregated when more than one failed, and +does nothing on a second call. The stack starts in 1.5 to 4.3 s and stops in under 0.4 s with no session attached +(measured 2026-10-20). + +**Not reproduced.** + +- The system-variables folder `_1` beside `root`: its node and browse name were not recorded. +- Four of the real server's seven endpoints: the demo offers one secured policy, not three. +- `Float`, `Int64` and `Byte` variables: the lattice yields doubles and the archive stores them, and a + rounded integer variable would differ from its archive row. +- `Bad` statuses. +- An advertised host name that differs from the dialled address: the real server advertises its + machine name, the demo binds and advertises `127.0.0.1`. +- The empty first notification under a monitored-item queue above 1: the stock stack delivers the + real value at any queue size. +- An abrupt server kill: the server offers `DisposeAsync`, a graceful stop, only. ## Thinning into the coarse layers @@ -158,6 +302,19 @@ image `semiplot-bench:test` with the session; the container, its volume and ever it. The images the run *pulled* — `ghcr.io/semiteq/semibase:latest` and the base image — stay, because re-pulling every run would put the registry on the critical path. +**The OPC UA demo server's tests need no container.** `DemoScadaServerLifecycleTests` and +`DemoScadaServerTests`, under `DemoScada/`, run in `DemoScadaCollection`, outside +`ArchiveDatabaseCollection`. The collection carries no fixture: it only keeps two servers from starting +side by side, and a test or a class owns its own server. The tests never dial 49875. `DemoScadaHost` +starts each server on a free port, read from a `TcpListener` on `127.0.0.1` port 0 and closed again, +because the stack asked for port 0 binds 4840; it skips 4840 and 49875, and retries up to three +attempts when another process takes the port first, recognised by the `BadNoCommunication` inner +exception, so a certificate failure surfaces on the first attempt. The host owns a temporary +certificate directory, which it deletes, and a `FakeLogCollector` of the server's `Warning`-and-above +records, which the assertion messages carry. It fixes `Asia/Tokyo` as the archive zone: no DST, and +nine hours off the UTC Linux runner, so a stamp that treats local time as UTC fails there. The busy-port +test holds its port with a `TcpListener` on `127.0.0.1`, the address the stack binds. + ## Where the provisioning comes from The provisioner is a layer of the bench image, not a binary on the machine. @@ -182,7 +339,7 @@ through, so a remote `DOCKER_HOST` that reports a hostname is not a supported se loader accepts only an IPv4 host. A missing container runtime is never a pass and never a skip: `InitializeAsync` lets the exception -through, and xunit fails every test of the collection with `TestPipelineException`. +through, and xunit fails every test of `ArchiveDatabaseCollection` with `TestPipelineException`. **`latest` is a moving tag on purpose.** Delivered installations update only the provisioner, so the pair worth testing is the newest `semibase` with the current reader. The image build pulls a `FROM` @@ -217,9 +374,20 @@ when the stand deletes `%TEMP%\SemiPlot\ConfigFiles` and then tries to open the against the missing folder, which is harmless (`overview.md#the-live-theme`). The JetBrains Aspire plugin (`me.rafaelldi.aspire`) is optional; it adds per-resource debugging (attaching to the converge job or the writer individually) on top of what the Aspire dashboard already shows. The AppHost injects the standard -OpenTelemetry and console-formatter environment variables into every project resource; neither the seeder -nor the viewer carries an OpenTelemetry SDK or the `Microsoft.Extensions.Logging` console provider, so -the variables are inert. +OpenTelemetry and console-formatter environment variables into every project resource. Neither the +seeder nor the viewer carries an OpenTelemetry SDK, and the viewer carries no +`Microsoft.Extensions.Logging` console provider. The seeder carries one for the OPC UA server's log, +built with `LoggerFactory.Create` and no configuration source, so the variables are inert in both +(`#the-opc-ua-demo-server`). + +The writer runs `--follow 1 --change-seconds 0.5 --opc-port 55875`, so the stand serves OPC UA at +`opc.tcp://127.0.0.1:55875`. The operator's machine runs Simple-Scada on 49875; a stand writer there +would refuse to start whenever the SCADA runs and stop the archive tail with it, which is the reason +the stand's Postgres sits on 55432 rather than 5432. Windows hands out outbound ports up to 65535 +(`netsh int ipv4 show dynamicport tcp`), so 55875, like 55432, can meet an outbound socket; a collision +fails the writer at once with the busy-port line. The stand passes no `--archive-lag` yet: the viewer +still takes its live edge from the archive poll, and a lagged archive would put the chart behind the +clock with nothing to fill the span. There is no volume: `converge` recreates the archive on every start regardless of what a previous session left, so a volume would carry nothing across runs. Every stand start pays `initdb`, @@ -323,14 +491,14 @@ What the server can be asked afterwards, which needs no screen: | Did it read history from the seeded span? | `idx_tup_fetch` on the partitions the fill landed in, `tpmd
` | | Did any read fall back to a sequential scan? | `seq_scan` on the same partitions, which `ExplainPlanTests` forbids | | Which failure did the operator get? | `%TEMP%\SemiPlot\Logs\semiplot.log` on the stand, `C:\DISTR\Logs\SemiPlot\semiplot.log` on an installation. A startup failure writes its error and a `[FTL]` line, except the two that happen before the logger exists: a bad launch argument and an unusable `--log-file` are reported in the failure window alone (`SemiPlot/SemiPlot.UI/Program.cs`) | -| Did the live edge reach the chart? | Run `--follow 1` against the same database and watch the chart with **Sticky** on; at `--logging-level debug` the log carries one realtime line per tick | +| Did the live edge reach the chart? | The stand's writer already runs `--follow 1 --opc-port 55875`; outside the stand, run `--follow 1 --opc-port ` against the same database. Watch the chart with **Sticky** on; at `--logging-level debug` the log carries one realtime line per tick | The failure states are forced from outside the application: stop the container for an unreachable server, rename `semiplot_tags` for an unfinished provisioning, change the password in the connection file for a refused login, delete the catalogue rows for an empty catalogue. Nothing machine-dependent lives in `AppHost.cs`: the role passwords are `BenchRoles`' public -constants, the port is 55432, the configuration and log paths come from `DemoDirectories`, and the +constants, the ports are 55432 for Postgres and 55875 for OPC UA, the configuration and log paths come from `DemoDirectories`, and the seeder and the viewer both take the machine's own time zone. The `Live demo` run configuration at the repository root's `.run/` is an `AspireHostConfiguration` over `SemiPlot.AppHost`'s `http` launch profile. diff --git a/docs/architecture/testing-strategy.md b/docs/architecture/testing-strategy.md index acd1b2c..8d615b7 100644 --- a/docs/architecture/testing-strategy.md +++ b/docs/architecture/testing-strategy.md @@ -35,7 +35,7 @@ and built from the commit under test. Its value is diagnosis. | --- | --- | | Navigation, scale, cursor geometry | `SemiPlot.Tests.Unit/Core/Trends/TrendNavigationModelTests.cs`, `PenScaleModelTests.cs`, `MinimapGeometryTests.cs`, `Chart/CursorReadoutModelTests.cs`, `DeltaCursorModelTests.cs` | | The seeder's generation rules | `SemiPlot.Tests.Unit/LayerThinnerTests.cs`, `RawLayerGeneratorTests.cs`, `BreakGenerationTests.cs`, `PartitionScriptTests.cs` | -| The demo writer's own rules and the command line | `SemiPlot.Tests.Unit/LiveTailGeneratorTests.cs`, `SharedLatticeTests.cs`, `SeederCommandTests.cs`. Its coarse flush is not here: it reads the finer layer back from the server and thins it with `LayerThinner.Thin`, so `Integration/CoarseFlushTests.cs` carries it, gated | +| The demo writer's own rules and the command line | `SemiPlot.Tests.Unit/LiveTailGeneratorTests.cs`, `SharedLatticeTests.cs`, `SeederCommandTests.cs`, `FollowEdgeTests.cs` (the archive edge's windows under `--archive-lag`). Its coarse flush is not here: it reads the finer layer back from the server and thins it with `LayerThinner.Thin`, so `Integration/CoarseFlushTests.cs` carries it, gated | | The error types' shape and extent arithmetic | `SemiPlot.Tests.Unit/Errors/DataErrorTests.cs`, `Data/ArchiveExtentTests.cs` | | The provider's statement text and its binder | `SemiPlot.Tests.Unit/Postgres/ArchiveStatementTextTests.cs` | | The bucketed fold: one column per bucket, the gap column, the DST drop | `SemiPlot.Tests.Unit/Postgres/BucketedRowFoldTests.cs`, `Postgres/RealArchiveGapTests.cs` | @@ -145,7 +145,7 @@ An integration test crosses exactly one boundary to a real implementation of som repository does not build, to verify the translation across that seam. A fake cannot do this job: it would encode our own assumption on both sides. The value is contract verification. -There are four families — the same category with different foreign parties. +There are five families — the same category with different foreign parties. **Against a real PostgreSQL**, in `SemiPlot.Tests.Integration/`: every class carrying `[Collection(ArchiveDatabaseCollection.Name)]` outside `Journeys/`, for example `PostgresHistoryReadTests`, @@ -178,6 +178,14 @@ listener). They cross the machine boundary and need no container, so they carry no `PostgresContainerFixture`; a missing container runtime does not fail them. `UnreachableArchive.Settings` gives them the unreachable address and an adjustable poll interval. +**Against the OPC Foundation stack** — `DemoScadaServerLifecycleTests` and `DemoScadaServerTests`, under +`SemiPlot.Tests.Integration/DemoScada/`, start the seeder's OPC UA demo server in process and read it +through a 2.0.1 client over loopback. They run in `DemoScadaCollection`, which carries no fixture and only +keeps two servers from starting side by side; they take no container and never dial 49875. Seams guarded: +the endpoints and the anonymous token policy, the namespace at index 3, the address space, the served +values and stamps under `Asia/Tokyo`, the start stamp, the busy-port mapping and the server's lifetime +(`bench.md#the-opc-ua-demo-server`). + **Against a real Avalonia** — `SemiPlot.Tests.Unit/UI/`, under `[AvaloniaFact]` and `[AvaloniaTheory]`, for example the classes that realise a view: `ChartPointerInputTests`, `MinimapPointerInputTests`, `MinimapViewTests`, `TrendChartViewTests`, `ChartMarkerViewTests`, `AxisScalePanelViewTests`, @@ -391,14 +399,14 @@ Nothing skips. `SemiPlot.Tests.Unit` runs on any machine with the SDK, and failure on both CI legs; in `SemiPlot.Tests.Integration` a missing container runtime throws out of `PostgresContainerFixture.InitializeAsync`, and xunit fails every test of the collection with `TestPipelineException`. Only the container collection fails as a whole: the socket classes carry no -collection and run without a runtime. +collection, and the `DemoScadaCollection` classes take no fixture, so both run without a runtime. CI has two jobs: `unit-windows` runs `SemiPlot.Tests.Unit` on `windows-latest`, the platform the viewer ships on, and `linux` builds the solution once on `ubuntu-latest` and runs both projects, the runner's Docker daemon serving the fixture. The Windows runner cannot host a Linux container and runs `SemiPlot.Tests.Integration` nowhere but the Linux leg. That includes `RealtimePollTests`, -`CatalogueReadUnreachableTests`, `RealtimeSubscriptionDisposalTests` and `ConvergeReadinessTests`: they -need no container, but they live in +`CatalogueReadUnreachableTests`, `RealtimeSubscriptionDisposalTests`, `ConvergeReadinessTests`, +`DemoScadaServerLifecycleTests` and `DemoScadaServerTests`: they need no container, but they live in Integration, so they run only on the Linux leg. ## Ownership diff --git a/docs/plans/completed/20261020-opc-demo-server.md b/docs/plans/completed/20261020-opc-demo-server.md new file mode 100644 index 0000000..5f616c8 --- /dev/null +++ b/docs/plans/completed/20261020-opc-demo-server.md @@ -0,0 +1,736 @@ +# OPC UA demo server in the demo writer + +## Overview + +- The seeder's demo writer (`--follow`) gains a Simple-Scada-shaped OPC UA server. Every pen the writer + archives is also a variable of that server, and each lattice change is served at its own instant, with + the `SourceTimestamp` the vendor's server would give it. Nothing new produces values: the server reads + the same lattice `LiveTailGenerator` writes, so every served value is an archive row; outside the two DST + transitions the stream and the archive's change rows agree row for row. +- A new `--archive-lag ` holds the archive inserts back by that span, so the archive trails the + OPC stream the way the SCADA's buffered archive trails the plant (roadmap, "Root cause"). +- A new `--opc-port ` names the listening port. A follow run requires it: the real Simple-Scada + listens on 49875 on the operator's machine, and a writer that took a default port would either fail + beside it or take the port from it. `SemiPlot.AppHost` passes the stand's port, 55875. +- The integration project starts the same server in process on a free port, so the next slice + (`opc-live-source`) can test its client without the real SCADA. +- The package line `OPCFoundation.NetStandard.Opc.Ua.*` 2.0.1 enters `Directory.Packages.props`, the four + central `Microsoft.Extensions.*` pins move from 10.0.8 to 10.0.12, which 2.0.1 requires, and + `Microsoft.Extensions.Logging.Console` 10.0.12 joins them for the server's warnings. +- No viewer code changes. The shipped viewer does carry `Microsoft.Extensions.DependencyInjection` and + `.Logging` 10.0.12 instead of 10.0.8, because `SemiPlot.UI` references both through the central pins. No + OPC client code enters a production project; the test-side client lives in `SemiPlot.Tests.Integration`. + +## Context (from discovery) + +- Roadmap: docs/plans/roadmaps/20261019-opc-live-edge-roadmap.md — slice opc-demo-server +- Verified against `master` at `54f455e`, read through `git show master:`. Line numbers are at that + commit. Scratch probes ran outside the repository on 2026-10-20: `srvprobe` (a 2.0.1 server and client + in one process, run on Windows and in the `mcr.microsoft.com/dotnet/sdk:10.0` Linux container) and + `pinprobe` (a `git archive master` copy with this slice's package changes, built and unit-tested). +- The demo writer: `Program.FollowAsync` reads the archive edge once through `StaleArchiveGuard`, then each + tick appends `LiveTailGenerator.Generate(options, edge, now)` and runs `CoarseFlush.FlushAsync(options, + edge, now)`, then sets `edge = now` (`SemiPlot/SemiPlot.Tools.ArchiveSeeder/Program.cs:46-89`). Ctrl+C + stops the loop where it waits (`:50-52`, `:84-88`). The clock is `DateTime.Now` with its `Kind` stripped + (`:93-96`). `ReportingAsync` prints `SeederException`, `NpgsqlException`, `ArgumentException` and + `FormatException` as one line and lets anything else through with its stack trace (`:16-29`). +- `FollowOptions(ConnectionString, Interval, PenCount, Seed, ChangeSeconds)` and `MaximumSeconds = 86400` + (`FollowOptions.cs:4-14`). Constructed by `SeederCommand.Interpret` (`SeederCommand.cs:322`) and in four + tests: `CoarseFlushTests.cs:244-252`, `FollowRestartTests.cs:80-83`, `LiveTailGeneratorTests.cs:142-145`, + `SharedLatticeTests.cs:28-33`; `SeederCommandTests.cs:65-96` asserts its fields. +- The command line: `--follow` selects the follow run (`SeederCommand.cs:62-68`); `_seedingOnly` lists the + options a follow run rejects (`:70-75`, checked at `:310-318`); there is no mirror list of options a + seeding run rejects. `ParseSeconds` refuses NaN, infinity, `<= 0` and `> MaximumSeconds` (`:217-238`). + `ParseWholeNumber` takes a minimum and maximum (`:175-199`). `converge` registers its own options + (`:146-162`), so an option not added to it is refused by System.CommandLine. +- The lattice: `RawLayerGenerator.AppendWindow` is the one emitter (`RawLayerGenerator.cs:53-96`), a change + sits at `index * intervalTicks` (`:74`), `FirstIndex` is the private ceiling (`:176-179`), + `ValueAt` is private (`:181-186`), `ChangeIntervalTicks` rounds to whole milliseconds (`:100-105`). + `LiveTailGenerator.Generate(options, after, to)` emits `after < t <= to` through `AppendWindow` + (`LiveTailGenerator.cs:7-32`). `SharedLatticeTests` pins one lattice for the seeding run and the follow + run (`SharedLatticeTests.cs:9-91`; `bench.md:42-51`). +- The pens: `RawLayerGenerator.SelectPens(count)` takes them round-robin across the groups of + `PrimaryGroup` (`RawLayerGenerator.cs:23-51`). `SyntheticPen.Name` is the catalogue name and + `PrimaryGroup` the first group, the empty string for an ungrouped pen (`SyntheticPen.cs:5-20`). + `UncommissionedPenId` 5001 has no group (`SyntheticPenCatalog.cs:102`), so `""` is a sixth round-robin + group and the default 8-pen slice is 1000, 2000, 3000, 4000, 5000, 5001, 1001, 2001. `TwoGroupPenId` 1000 + also sits in `Watchlist` (`SyntheticPenCatalog.cs:92`). `converge` seeds `SeederOptions` defaults (8 pens) + and the AppHost's writer passes no `--pens`, so the stand's writer and catalogue hold the same 8 pens + (`bench.md:249-251`, `AppHost.cs:50-52`). +- The time boundary: the archive holds naive local time (`Program.cs:91-96`); `ArchiveTimeConverter` in + `SemiPlot.DataSource.Postgres` is the viewer's one conversion, `ToUtc` resolving a skipped local time by + the base offset and `ToArchiveLocal` the reverse (`ArchiveTimeConverter.cs:7-37`). + `ArchiveTimeConverterTests` fixes `Europe/Berlin`, whose 2026 rule matches on Windows and on ubuntu + (`SemiPlot/SemiPlot.Tests.Unit/Postgres/ArchiveTimeConverterTests.cs:9-17`). The seeder references `SemiPlot.Core` only + (`SemiPlot.Tools.ArchiveSeeder.csproj:7-14`). +- The guard on a restart: `StaleArchiveGuard.CheckAsync(connection, now)` refuses an archive more than + five minutes behind `now` (`StaleArchiveGuard.cs:11`, `:19-44`). +- The pins: `Microsoft.Extensions.DependencyInjection`, `.DependencyInjection.Abstractions`, `.Logging`, + `.Logging.Abstractions` at 10.0.8, `Microsoft.Extensions.Diagnostics.Testing` at 10.6.0 + (`SemiPlot/Directory.Packages.props:19-23`). `SemiPlot.Tests.Unit`, `SemiPlot.Tests.Integration` and + `SemiPlot.UI` reference `DependencyInjection` and `Logging` directly; `SemiPlot.DataSource.Postgres` + references the two `Abstractions`. `TreatWarningsAsErrors` and `AnalysisMode Recommended` hold for every + project (`SemiPlot/Directory.Build.props:10-16`). +- The AppHost: the writer runs `--connection ... --follow 1 --change-seconds 0.5` after `converge` + (`AppHost.cs:50-52`); the stand's Postgres sits on host port 55432 so it never meets a local server on + 5432 (`AppHost.cs:18`, `:34`); argument constants are strings (`AppHost.cs:21`); the AppHost cannot + compile against the seeder, so bench constants repeat there (`AppHost.cs:8-9`, `bench.md:280-283`). The + AppHost injects console-formatter environment variables that the seeder ignores (`bench.md:219-222`). +- The integration project: no `[Collection]` and no fixture means no container, as + `ConvergeReadinessTests` runs (`ConvergeReadinessTests.cs:12-16`). It references the seeder, the provider, + Core and the UI (`SemiPlot.Tests.Integration.csproj:29-34`), carries no `xunit.runner.json`, and runs on + the Linux CI leg only (AGENTS.md, Test). +- Configuration: `ConfigFiles/` holds `app/` and `connection/` only. `DeliveredConfigurationTests` names + its loaders one by one (`DeliveredConfigurationTests.cs:29-85`). The installer copies `ConfigFiles\*` + recursively with `onlyifdoesntexist` (`Installer/SemiPlot.iss:71`). +- The real server, measured 2026-10-19 (roadmap table, lines 41-54): namespace 3 is + `http://simple-scada.com/UA`; `Objects` holds folder `ns=3;s=root_` with browse name + `root`; a variable is `ns=3;s=` with browse name = the variable name; a SCADA group + is a folder beside `root`, `ns=3;s=_`, and a variable inside keeps its address; + `None` + Anonymous is offered beside Sign and SignAndEncrypt; the initial value arrives at once with the + stamp of the last change; after a start every first notification carries the start instant; a + monitored-item queue above 1 makes the first notification an empty `Good` value. Support states + `ns=3;s=` is stable (`sources.md:63`). +- The 2.0.1 client spike connected to the real server with an anonymous token policy whose + `SecurityPolicyUri` is null (`dotnet-spike/v20/Program.cs`, `SPIKE_TOKENPOLICY` unset, runs in + `dotnet-spike/run20/`). The client accepts the server's policy only when token type and + `SecurityPolicyUri` match, null equal to empty (`Session20.cs:6180-6195` of the stack source). +- `GetEndpoints` of the real server, read 2026-10-20 with asyncua (`scratchpad/opc_assumptions.py`): + seven endpoints, all advertising `opc.tcp://WORKING-PC:49875`: `None`, and `Sign` plus `SignAndEncrypt` + under each of `Aes128_Sha256_RsaOaep`, `Basic256Sha256` and `Aes256_Sha256_RsaPss`. Every endpoint + carries one token policy, `PolicyId = Anonymous`, type Anonymous, `SecurityPolicyUri = + http://opcfoundation.org/UA/SecurityPolicy#None`. So the real server advertises `#None`, not an empty + URI, and the spike's null-policy client still connected to it. +- The same read of `Objects`: `Server`, then `FD34514A35_1` (the system-variables folder), + `FD34514A35root_2` (browse name `root`), then `FD34514A35group1_3` (a user group). Every browse name, + the folders' and the variables' (`Fast1`, `Fast2`, ...), carries namespace index 3. Namespace array: + `[http://opcfoundation.org/UA/, urn:WORKING-PC:SimpleScada:UAServer, + http://www.simple-scada.com/UA/Namespace, http://simple-scada.com/UA]`. + +### Measured on 2026-10-20 (scratch probes, not committed) + +Every row holds on Windows and in the Linux container unless it names one platform. + +| Question | Answer | +| --- | --- | +| Does a 2.0.1 server bind port 0 to a free port? | No. `BaseAddresses = opc.tcp://127.0.0.1:0` listened on 4840, the OPC UA default | +| Which interface does it bind? | `127.0.0.1` only: a connect to `::1` and to every non-loopback address of the host fails (Windows: `netstat` shows `127.0.0.1: LISTENING`) | +| Does a `None`-only server run without an application certificate? | No. `ValidateAsync(ApplicationType.Server)` throws "ApplicationCertificate must be specified", and every `CreateSession` fails with a `NullReferenceException` inside the server | +| What does the certificate cost? | `ApplicationInstance.CheckApplicationInstanceCertificatesAsync(true, null, ct)` over directory stores writes three files (`own/certs`, `own/private`, `trusted/certs`) in 124-304 ms | +| Can the stores live in memory? | Not on the non-DI path: store type `InMemory` throws "Invalid store type name" at validation; the provider registers only through `CertificateManagerOptions.StoreProviders` | +| What does the server advertise for Anonymous? | `SecurityPolicyUri = ...#None` on the `None` endpoint, the same as the real server there; a client asking with a null policy then fails `BadSecurityChecksFailed`. The plan keeps `#None` and has the client take the advertised policy instead (Solution Overview, Security) | +| Sign and SignAndEncrypt beside None? | Two more `ServerSecurityPolicy` entries (`Basic256Sha256`) add the two endpoints; the null-policy anonymous client still connects over `None` | +| Which index does the custom namespace get? | 2, with `http://opcfoundation.org/UA/Diagnostics` at 3. `server.NamespaceUris.GetIndexOrAppend()` inside `INodeManagerFactory.Create`, ahead of the manager's own registration, puts Diagnostics at 2 and `http://simple-scada.com/UA` at 3 | +| What else does a default server do? | It dials a discovery server at `opc.tcp://localhost:4840` (`RegisterServer2`) until `ServerConfiguration.MaxRegistrationInterval = 0` | +| A busy port? | `StartAsync` throws `ServiceResultException` `BadNoCommunication`, "Failed to establish tcp listener sockets on port N" | +| Start and stop cost | `StartAsync` 1.5-4.3 s. `StopAsync` 141-318 ms with no session, 5.0-5.1 s with a session attached. A second server on the same port starts 84 ms (Windows) and 97 ms (Linux) after the first stopped with a session attached | +| Two servers in one process | Start and serve side by side on two ports | +| The initial value and changes | Delivered at subscribe with the `Timestamp` the variable was created with; with `SamplingInterval = 0`, `ClearChangeMasks` after a value and timestamp write raised one notification per write every 0.5 s; an unknown node fails its monitored item with `BadNodeIdUnknown` | +| The pins | With the two OPC packages at 2.0.1 and the pins at 10.0.8, restore fails `NU1605` (package downgrade) in `SemiPlot.Tests.Unit` and `SemiPlot.Tests.Integration` for `DependencyInjection` and `Logging`. With the four pins at 10.0.12 the solution builds with 0 errors (the two known `NU1902` of the AppHost remain) and the unit suite passes 2297 of 2297 | +| `Diagnostics.Testing` | Stays 10.6.0: the client pulls `Microsoft.Extensions.Telemetry.Abstractions` 10.10.0 into `SemiPlot.Tests.Integration`, and a `FakeLogger` 10.6.0 logs and reads back over it | +| Analyzers in the packages | None: `Opc.Ua.Server` and `Opc.Ua.Core` 2.0.1 ship no `analyzers/` folder | + +## Development Approach + +- **testing approach**: Regular (code first, then tests), with one exception: the `SharedLatticeTests` fact + of Task 3 is confirmed red against a deliberately wrong publisher before it is kept. +- complete each task fully before moving to the next +- make small, focused changes +- **CRITICAL: every task MUST include new/updated tests** for code changes in that task +- **CRITICAL: all tests must pass before starting next task** - no exceptions +- **CRITICAL: update this plan file when scope changes during implementation** +- run tests after each change +- compatibility: a follow run now requires `--opc-port` and starts a server; a hand command that omits the + option fails before anything connects, naming it. Seeding and `converge` runs are unchanged. + +## Testing Strategy + +- **unit tests** (`SemiPlot.Tests.Unit`, plain `[Fact]`, traits `Component=Core`, `Area=Data`, + `Category=Unit`): the command line, the lattice lookup, the follow edge, the shared lattice. +- **integration tests** (`SemiPlot.Tests.Integration`, plain `[Fact]`, traits `Component=Core`, + `Area=Data`, `Category=Integration`): the server against a test-side 2.0.1 client over loopback. They + share one named collection, `DemoScadaCollection`, with no collection fixture, so they never run beside each + other + and need no container; they stay outside `ArchiveDatabaseCollection` and never dial 49875. +- **time zone**: the CI Linux leg runs in UTC, where a local-as-UTC stamp passes unnoticed. The value test + runs the server under `Asia/Tokyo` (+09:00, no DST, the same IANA id on Windows and on ubuntu) and checks + every stamp against true UTC; a zone without DST keeps the test clear of the fall-back hour, and a + local-as-UTC stamp misses by nine hours. +- **real time**: the server publishes on the wall clock. The tests use short change intervals and assert + values, stamps and a floor on the notification count, never latency. +- **server logs in tests**: every server gets a logger factory over a `FakeLogCollector` owned by its host + (the class fixture or the test), and every assertion message carries the records of `Warning` and above + added since that test began, so a failing test shows the server's side. +- no e2e suite exists; the stand is checked by hand (Post-Completion). + +## Acceptance Evidence + +Today the stand serves no OPC UA: `Test-NetConnection 127.0.0.1 -Port 55875` reports +`TcpTestSucceeded : False` while the stand runs, and the archive's newest row follows the clock within one +tick. + +Automatable, after the slice: + +1. `dotnet test SemiPlot/SemiPlot.Tests.Integration/SemiPlot.Tests.Integration.csproj --filter "FullyQualifiedName~DemoScada"` + passes with no container runtime: three endpoints with `None` among them and one anonymous token per + endpoint, `#None` on the `None` endpoint (the stock stack leaves the secured endpoints' URI empty; Task 4 + ⚠️); the namespace `http://simple-scada.com/UA` at index 3; + `root_2` ahead of the group folders numbered from 3 under `Objects`; every one of the stand's 8 pens served as `ns=3;s=` with its catalogue + name; every notification after the first carries the lattice value at a true-UTC lattice instant under + `Asia/Tokyo`; the first notification after a start carries the start instant; an unknown node fails + `BadNodeIdUnknown`; a busy port fails with a `SeederException` naming it; a stopped server starts again + on its port; a second start reuses the certificate. +2. `dotnet test SemiPlot/SemiPlot.Tests.Unit/SemiPlot.Tests.Unit.csproj --filter "FullyQualifiedName~SharedLatticeTests|FullyQualifiedName~SeederCommandTests|FullyQualifiedName~LiveTailGeneratorTests|FullyQualifiedName~FollowEdgeTests"` + passes; `SharedLatticeTests` holds three facts, the new one red against a publisher that reads the next + lattice index. +3. `dotnet build SemiPlot.slnx` at 0 errors and no warning beyond the AppHost's two `NU1902`; + `dotnet test SemiPlot.slnx` green on both legs of CI. + +Manual, on the stand: see Post-Completion. + +## Progress Tracking + +- mark completed items with `[x]` immediately when done +- add newly discovered tasks with ➕ prefix +- document issues/blockers with ⚠️ prefix +- update plan if implementation deviates from original scope + +## Solution Overview + +Two types own the server in `SemiPlot.Tools.ArchiveSeeder`: `DemoScadaServer` (configuration, start, +stop, the publishing loop) and `DemoScadaNodeManager` (the address space and the variable writes). The +server starts from the follow run's options, builds the address space for +`RawLayerGenerator.SelectPens(options.PenCount)`, and runs one publishing loop. The loop wakes at each +lattice change instant, asks `LiveTailGenerator` for the change rows at that instant, and writes them into +the variables. The archive writer keeps its statements; its window end moves back by `--archive-lag`, +through a new `FollowEdge` type that owns the edge. + +Decisions: + +- **[decision] One source of values.** The server publishes only rows `LiveTailGenerator` emits. A new + `LiveTailGenerator.ChangesAt(options, instant)` returns the change rows at the newest lattice instant at + or before `instant`, through `Generate(options, change.AddTicks(-1), change)`, so `AppendWindow` stays the + one emitter; a window one tick wide holds the change row and never its anchor, whose instant lies 100 ms + earlier. The newest instant comes from `RawLayerGenerator.LatestChangeTicks`, which sits beside + `FirstIndex` so the lattice arithmetic stays in one file. `SharedLatticeTests` gains the third path. +- **[decision] The server publishes changes, not anchors.** The anchor row repeats the previous value one + poll interval early; it is the archive's two-rows-per-change shape, not a change the plant made. +- **[decision] One value per pen per wake.** A variable holds one current value and the client's monitored + item keeps a queue of 1. A process stall that skips lattice instants publishes the newest one only; the + archive still receives every row. +- **[decision] The start stamp.** At start each variable takes the lattice value at the newest change at or + before the start and the start instant as its `SourceTimestamp`; the loop's first publish is the next + change. The loop's last served instant is seeded with `LatestChangeTicks` of the start, so its first write + is the next change and no stamp ever moves back past the start. A pen that has not changed since the + start keeps the start stamp, as on the real server. The start instant is read ahead of the stack's + 1.5-4.3 s start, so a change inside the start is written as soon as the loop runs: a client sees the + start stamp only when it subscribes before the first change after the start, never at the stand's 0.5 s + lattice. Seeding the loop from the clock after the start instead would skip such a change until the + next one, a day of stale value at an 86400 s lattice. +- **[decision] The time boundary reuses the viewer's converter.** The seeder gains a project reference to + `SemiPlot.DataSource.Postgres` and the server takes a `TimeZoneInfo`, wrapped in `ArchiveTimeConverter`. + `SourceTimestamp = converter.ToUtc(row.Timestamp)` and the loop's clock is + `converter.ToArchiveLocal(DateTime.UtcNow)`. A live point and the archive row that later replaces it then + map to one UTC instant by construction, through the code the viewer runs. Production passes + `TimeZoneInfo.Local`; the integration tests pass `Asia/Tokyo`. A second copy of the rule inside the + seeder would be a duplicate the next DST bug splits. The cost is the autumn fall-back hour: the local + clock repeats 02:00-03:00, `LatestChangeTicks` falls behind the last served instant, and the server + publishes nothing new for that hour, while its first-pass stamps resolve to standard time, an hour ahead + of true UTC. The archive rows of that hour take the same path, so the two agree with each other; the real + server does not stall there. At the spring jump the archive tick spans the skipped hour and writes lattice + rows at local instants that never existed (today's behaviour), while the server publishes only the + instant after the jump. +- **[decision] Address space.** Namespace `http://simple-scada.com/UA` is pinned at index 3 by registering + the Diagnostics URI first (measured above): the real server has it there and the vendor calls + `ns=3;s=` stable, so this is the real contract, not a demo convenience. The + project ID is a private constant `5E4D0C1A0B`, ten hex characters like the measured `FD34514A35` and + different from it, so a client that hard-codes the real ID fails against the demo. `Objects` organizes, + in this order, the folder `root_2` (browse name `root`) and then one folder per distinct + non-empty `PrimaryGroup` of the served pens in `SelectPens` order, `_` with `n` from + 3, as the real server numbers them (measured above; the real `_1` is its system folder, not reproduced). + Every browse name carries namespace index 3 (measured above). A client finds `root` by browse name, never + by position or by a fixed node id. `root` organizes every served pen; each group folder organizes the same + variable nodes of its pens. An ungrouped pen (5001) sits under `root` only, and a pen's later groups + (`Watchlist`) get no folder: the catalogue groups stand in for SCADA groups, and only the primary one + is placed. Variables: ``, browse and display name `SyntheticPen.Name`, `DataType` Double, + scalar, current-read access, status `Good`. +- **[decision] Not reproduced.** The `Sys*` sibling folder (its node and browse name were not recorded); + `Float`/`Int64`/`Byte` variables (the lattice yields doubles and the archive stores them; a rounded integer + variable would differ from its archive row); `Bad` statuses; the advertised host name differing from the + dialled address (the demo binds and advertises `127.0.0.1`); the empty first notification under a + monitored-item queue above 1 (the stock stack delivers the real value at any queue size); an abrupt + server kill (the server offers `DisposeAsync`, a graceful `StopAsync`, only). `opc-live-source` covers + numeric conversion, `Bad`, the queue size of 1 and keeping the configured address over the advertised + one in unit tests of its own. It reproduces a project restart, which destroys the session, by disposing + the demo server and starting a new one on the same port (measured at under 100 ms); a loopback TCP + relay its test cuts is the second case, a lost transport with the session still alive on the server. +- **[decision] Security.** Three endpoints, `None`, `Sign` and `SignAndEncrypt`, the last two under + `Basic256Sha256`, one of the three secured policies the real server offers (measured above); the endpoint + list exists so a client that picks a secured endpoint fails here as it would on the real server. One + stock anonymous token policy and no `GetUserTokenPolicies` override: the stack advertises it on `#None` at + the `None` endpoint, as the real server does, and with an empty `SecurityPolicyUri` at `Sign` and + `SignAndEncrypt`, where the real server shows `#None` (measured, Task 4 ⚠️). A test client takes the anonymous token policy from + the `None` endpoint's own `UserIdentityTokens` rather than building a null-policy one, which works against + both servers; if a 2.0.1 client still fails against the demo, record the error and the client + configuration in the plan and stop that task rather than diverging from the real server's + advertisement. Client certificates are never trusted, so only `None` admits a session. + `MaxRegistrationInterval = 0` stops the discovery-server registration. +- **[decision] The application certificate lives in a directory the caller names.** The server cannot run + without one (measured), and the in-memory store needs the DI hosting path. The follow run passes the + fixed `%TEMP%\SemiPlot\OpcDemoServer` (`Path.Combine(Path.GetTempPath(), "SemiPlot", "OpcDemoServer")`), + so every run, killed or stopped, reuses one certificate. The first run creates the directory and nothing + removes it: it holds the server's certificate and private key, and `rejected/certs` collects the + certificate of every client refused on a secured endpoint. `DemoDirectories` sweeps only `ConfigFiles` + and `Logs` beside it. Tests pass a directory of their own and delete it. + ASSUMPTION: `CheckApplicationInstanceCertificatesAsync` replaces a certificate that has expired; the + stand check of Post-Completion does not reach that case. +- **[decision] Server logging.** The stack logs through a `LoggerFactory` over the console provider, + `Warning` and above, single line, all of it to standard error (`LogToStandardErrorThreshold = + Warning`), so a rejected session leaves a line on the stand. The factory is built with + `LoggerFactory.Create` and no configuration source, so the console-formatter variables the AppHost + injects stay inert. Each session over the demo's `127.0.0.1` certificate logs one host-name line, + `Opc.Ua.CertificateValidationCore` "The domain 'opc.tcp://127.0.0.1:/' is not listed in the server + certificate", at `Error`, not `Warning` (measured 2026-10-20, Task 8 ⚠️); the `Warning`+ factory prints it, + and nothing suppresses or filters it. The line is expected. +- **[decision] The loop never ends on its own.** `DemoScadaServer.Completion` completes successfully only + when `DisposeAsync` cancels it, and a fault in the loop faults it; the loop catches its own token's cancellation and returns, so a graceful stop + leaves `Completion` completed successfully. `FollowAsync` checks it on every tick: a fault is rethrown, and a completion + without a fault is an `InvalidOperationException` ("the OPC UA publishing loop ended"), so a writer never + keeps appending beside a dead server. `DisposeAsync` attempts the loop cancellation, `StopAsync` and the + stack's disposal each, and rethrows only a real fault, aggregated when more than one failed. +- **[decision] Ports.** `--opc-port` is a whole number 1-65535 and required for a follow run, with no + default. Port 0 is refused at the command line, because the stack would bind 4840 instead of a free + port; `StartAsync` carries no guard of its own, since only the command line and the tests call it. A busy + port is a `SeederException` that names the port and says a running Simple-Scada holds 49875 by default. +- **[decision] The stand uses port 55875.** The operator's machine runs Simple-Scada on 49875; a stand + writer there would refuse to start whenever the SCADA runs, stopping the archive tail with it. The stand + already moves Postgres to 55432 for the same reason. The AppHost passes `--opc-port 55875`. This + machine's dynamic port range is 1024-65535 (`netsh int ipv4 show dynamicport tcp`, 2026-10-20), so no + port is clear of an outbound socket by range; 55875 carries the same small exposure 55432 does, and a + collision fails the writer at once with the busy-port line rather than serving nothing quietly. +- **[decision] Tests pick a free port.** A `TcpListener` on `127.0.0.1` port 0 reads a free port and + closes; the test helper starts the server there and retries with a fresh port, up to three attempts, when + the start fails with `SeederException` for a busy port, which covers another process taking the port in + between. The busy-port test holds its port with a `TcpListener` on `127.0.0.1`, the address the stack + binds. +- **[decision] `--archive-lag`.** A finite number of seconds, 0 to `FollowOptions.MaximumSeconds`, default + 0 (today's behaviour). `FollowEdge` owns the archive's edge: each tick it returns the window + `(edge, now - lag]` and moves the edge to its end, or returns nothing and keeps the edge while the + lagged clock has not passed it, so the edge never moves back (moving it back would regenerate rows the + archive holds and fail the `COPY` on the primary key). An empty archive seeds the edge at the lagged + clock. `StaleArchiveGuard` reads the lagged clock, so a restart compares the archive's edge with the + instant the archive should have reached, and the lag needs no bound below its five minutes. A restart + that drops a lag the previous run used meets an archive behind the unlagged clock; the guard's refusal + therefore names passing the previous run's `--archive-lag` beside `converge`, which drops the database. + `CoarseFlush` runs over the same window, because its opening-row insert reads the raw rows just written. + The OPC server is not lagged. +- **[decision] A graceful stop writes the held-back span.** On Ctrl+C the run appends and thins the + window `FollowEdge.Advance(LocalNow())` returns once before it exits, as the SCADA flushes every buffer on a + graceful project stop; a killed writer loses that span, as a killed SCADA does (roadmap line 54). +- **[decision] The AppHost passes no `--archive-lag` in this slice.** The stand's live edge still comes + from the archive poll until `opc-live-source`, and a lagged archive would put the stand's chart minutes + behind the clock with nothing yet to fill the span. `opc-live-source` adds the lag to the AppHost. +- **[decision] `ConfigFiles/opc/opc.yaml` arrives with `opc-live-source`, not here.** The loader that + reads it arrives there, and `DeliveredConfigurationTests` names its loaders one by one; a tracked file + created here would ship in any release cut between the slices with nothing to read or guard it, and a + guard written here would parse the viewer's file a second way. The tracked file carries 49875, the + plant's port; the stand's copy must carry 55875, which `opc-live-source` writes the way `converge` + writes 55432 into `connection/connection.yaml`. + +## Technical Details + +### Command line + +| Option | Run | Value | Default | +| --- | --- | --- | --- | +| `--archive-lag` | follow only | seconds, finite, `0 <= x <= 86400` | `0` | +| `--opc-port` | follow only, required there | whole number `1..65535` | none | + +A seeding run that names either one fails with `Option '' belongs to a follow run: ...` and exit 1, +mirroring the existing `_seedingOnly` rejection. A follow run without `--opc-port` fails with `Option +'--opc-port' is required for a follow run: ...`, naming 49875 as the port a running Simple-Scada holds. +`converge` does not register either option, so System.CommandLine refuses them there. `FollowOptions` +becomes: + +```csharp +public sealed record FollowOptions( + string ConnectionString, + TimeSpan Interval, + int PenCount, + long Seed, + double ChangeSeconds, + TimeSpan ArchiveLag, + int OpcPort) +``` + +No default values on the record: every construction site states both. + +### The server + +```csharp +public sealed class DemoScadaServer : IAsyncDisposable +{ + public static Task StartAsync( + FollowOptions options, + TimeZoneInfo archiveZone, + string certificateDirectory, + ILoggerFactory loggerFactory); + + public string EndpointUrl { get; } // opc.tcp://127.0.0.1:, printed by the follow run + public Task Completion { get; } // the publishing loop; see "The loop never ends on its own" + public ValueTask DisposeAsync(); +} +``` + +Review phase 1 removed `VariableCount`, a pass-through of `options.PenCount` that the follow run now prints +itself, and the `CancellationToken`, which no production caller passed (Task 6 gives the start no Ctrl+C +token). + +Files: `DemoScadaServer.cs` holds the server class and the publishing loop; it uses the stock +`StandardServer` unless the node-manager registration needs a subclass, and overrides no token policy. `DemoScadaNodeManager.cs` holds the `CustomNodeManager2` +subclass and its private `INodeManagerFactory`, which registers the Diagnostics URI ahead of the manager. +The namespace URI and the project ID are private constants of the node manager; the tests repeat both +literals, which pins the vendor-shaped values against an edit. + +`StartAsync` order: build the `ApplicationConfiguration` (server type, application name +`SemiPlotDemoScada`, URI `urn:localhost:SemiPlotDemoScada`, base address `opc.tcp://127.0.0.1:`, the +three security policies, one anonymous `UserTokenPolicy`, `MaxRegistrationInterval = 0`, directory stores +`own`, `issuer`, `trusted` and `rejected` under `certificateDirectory`), `ValidateAsync`, +`CheckApplicationInstanceCertificatesAsync(silent: true, ...)`, `AddNodeManager`, `StartAsync`, start the +loop. A `ServiceResultException` with `BadNoCommunication` becomes the `SeederException`. Every failure +from the stack's start up to the node manager's lookup disposes the stack before it propagates. + +Publishing loop, each wake: + +1. `now = converter.ToArchiveLocal(DateTime.UtcNow)` +2. `change = LatestChangeTicks(now)`; if `change` is not newer than the last served instant, skip to 4 +3. write `ChangesAt(options, change)` into the variables under the manager's `Lock`: value, `Timestamp = + converter.ToUtc(row.Timestamp)`, then `ClearChangeMasks` +4. wait until `change + interval`, recomputed from the clock on every wake + +### The follow run + +```csharp +public sealed class FollowEdge(DateTime edge) +{ + // The window (edge, end], moving the edge to end; null, edge kept, when end is not past the edge. + public (DateTime After, DateTime To)? Advance(DateTime end); +} +``` + +The archive's clock is `LocalNow() - options.ArchiveLag`. `FollowAsync` order: `using var` the console +logger factory, so its queue flushes after the server is gone; `StaleArchiveGuard.CheckAsync(connection, +archive clock)`; `new FollowEdge(newest ?? archive clock)`; start the server over +`TimeZoneInfo.Local`, the fixed certificate directory and that factory; report the plan, +including `archive lag s` and `opc , variables`; then the tick loop. +Each tick: the `Completion` check; `Advance(archive clock)`; on a window, append and flush over it as +today. After Ctrl+C: `Advance(LocalNow())`, the held-back span, appended and flushed without the token, +then the server and the logger factory are disposed in that order, and `stopped` prints between the two. +The server is disposed explicitly rather than by `await using`: when the tick loop throws, the run disposes +the server and rethrows the loop's exception, and a dispose fault beside it is only logged through the +factory, so the archive failure is the one the operator reads. + +### Project and package changes + +| File | Change | +| --- | --- | +| `SemiPlot/Directory.Packages.props` | add `OPCFoundation.NetStandard.Opc.Ua.Client` and `.Server` at 2.0.1 and `Microsoft.Extensions.Logging.Console` at 10.0.12; the four pins at `:19-20`, `:22-23` to 10.0.12; `Diagnostics.Testing` stays 10.6.0 | +| `SemiPlot.Tools.ArchiveSeeder.csproj` | `PackageReference` `OPCFoundation.NetStandard.Opc.Ua.Server` and `Microsoft.Extensions.Logging.Console`; `ProjectReference` `SemiPlot.DataSource.Postgres` | +| `SemiPlot.Tests.Integration.csproj` | `PackageReference` `OPCFoundation.NetStandard.Opc.Ua.Client` | + +The Server package brings `Opc.Ua.Configuration` (`ApplicationInstance`) transitively. Test files that use +the client alias `Opc.Ua.Client.Subscription` and `Opc.Ua.Client.MonitoredItem`, which collide with the +server's types of the same names once both assemblies are referenced (measured). + +## What Goes Where + +- **Implementation Steps**: code, tests and docs inside this repository. +- **Post-Completion**: the stand smoke check and the hand-offs to later slices. + +## Implementation Steps + +### Task 1: Pin the 2.0.1 package line + +**Files:** +- Modify: `SemiPlot/Directory.Packages.props` +- Modify: `SemiPlot/SemiPlot.Tools.ArchiveSeeder/SemiPlot.Tools.ArchiveSeeder.csproj` +- Modify: `SemiPlot/SemiPlot.Tests.Integration/SemiPlot.Tests.Integration.csproj` + +- [x] add both OPC packages at 2.0.1 and the seeder's and the integration project's references, leave the + pins at 10.0.8, run `dotnet restore SemiPlot.slnx` and confirm the `NU1605` downgrade errors in both + test projects (the guard fires at HEAD) +- [x] raise the four `Microsoft.Extensions.*` pins to 10.0.12; leave `Diagnostics.Testing` at 10.6.0 +- [x] add `Microsoft.Extensions.Logging.Console` 10.0.12 centrally and to the seeder, and the seeder's + `ProjectReference` to `SemiPlot.DataSource.Postgres` +- [x] `dotnet build SemiPlot.slnx`: 0 errors, no warning beyond the AppHost's two `NU1902` +- [x] run tests: `dotnet test SemiPlot/SemiPlot.Tests.Unit/SemiPlot.Tests.Unit.csproj` - must pass before + task 2 (no new test: the build and the unchanged suite are this task's evidence) + +### Task 2: The follow-only options + +**Files:** +- Modify: `SemiPlot/SemiPlot.Tools.ArchiveSeeder/FollowOptions.cs` +- Modify: `SemiPlot/SemiPlot.Tools.ArchiveSeeder/SeederCommand.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/SeederCommandTests.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/LiveTailGeneratorTests.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/SharedLatticeTests.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Integration/CoarseFlushTests.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Integration/FollowRestartTests.cs` + +- [x] add `ArchiveLag` and `OpcPort` to `FollowOptions` +- [x] add `--archive-lag` (a seconds parser that accepts 0, sharing `ParseSeconds`' NaN and infinity rule) + and `--opc-port` (`ParseWholeNumber`, 1 to 65535) to the root command, both follow-only, `--opc-port` + required in `Interpret`'s follow branch +- [x] add the mirror of `_seedingOnly`: a seeding run naming either option fails naming it +- [x] update the four test construction sites with `TimeSpan.Zero` and a port literal (the port is never + bound by these tests) +- [x] tests: a follow run with `--opc-port 55875` carries lag 0 and that port; a follow run accepts + `--archive-lag 120`; `--archive-lag 0` is accepted +- [x] tests: a follow run without `--opc-port` is refused naming the option; `--archive-lag -1`, `NaN`, + `86401`, `x` and `--opc-port 0`, `65536`, `8.5` are refused naming the value; a seeding run with + either option is refused naming the option; a `converge` run with its required set + (`SeederCommandTests.cs:226-227`) plus `--opc-port 1` is refused naming `--opc-port` +- [x] update the follow cases of `SeederCommandTests.cs:65-109` that run without `--opc-port` to pass it +- [x] run tests - must pass before task 3 + +### Task 3: The lattice point the server reads + +**Files:** +- Modify: `SemiPlot/SemiPlot.Tools.ArchiveSeeder/RawLayerGenerator.cs` +- Modify: `SemiPlot/SemiPlot.Tools.ArchiveSeeder/LiveTailGenerator.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/LiveTailGeneratorTests.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Unit/SharedLatticeTests.cs` + +- [x] add `internal static long LatestChangeTicks(long atTicks, long intervalTicks)` beside `FirstIndex`: + the newest change instant at or before `atTicks`, never below index 1 +- [x] add `LiveTailGenerator.ChangesAt(FollowOptions options, DateTime instant)`: one change row per + selected pen at `LatestChangeTicks`, through `Generate(options, change.AddTicks(-1), change)` +- [x] tests: `ChangesAt` at an exact lattice instant returns that instant; between instants returns the + earlier one; one row per pen in `SelectPens` order; no anchor row at intervals above and below the + 100 ms poll interval (1 s and 0.05 s) +- [x] `SharedLatticeTests`: a third fact - over one hour of a 0.5 s lattice, `ChangesAt` at every lattice + instant equals the follow window's change row at that timestamp, value and pen; confirm it fails + against a `ChangesAt` that reads index + 1, then restore +- [x] run tests - must pass before task 4 + +### Task 4: Start and stop the demo server + +**Files:** +- Create: `SemiPlot/SemiPlot.Tools.ArchiveSeeder/DemoScadaServer.cs` +- Create: `SemiPlot/SemiPlot.Tools.ArchiveSeeder/DemoScadaNodeManager.cs` (the factory and an empty address + space; Task 5 fills it) +- Create: `SemiPlot/SemiPlot.Tests.Integration/DemoScada/DemoScadaCollection.cs` +- Create: `SemiPlot/SemiPlot.Tests.Integration/DemoScada/DemoScadaHost.cs` +- Create: `SemiPlot/SemiPlot.Tests.Integration/DemoScada/OpcTestClient.cs` +- Create: `SemiPlot/SemiPlot.Tests.Integration/DemoScada/DemoScadaServerLifecycleTests.cs` + +- [x] implement `DemoScadaServer.StartAsync` and `DisposeAsync` as in Technical Details: configuration, + certificate directory, the three policies, the stock anonymous token policy, the namespace order, the + busy-port mapping, the loop start, `Completion` +- [x] test helpers: `DemoScadaCollection` (a named collection with no collection fixture); `DemoScadaHost.StartAsync` + picks a free port, retries a busy one up to three times, and owns a temporary certificate directory and + a `FakeLogCollector`; `OpcTestClient` opens a `None` + Anonymous session with the anonymous token policy + the `None` endpoint advertises and no client certificate, reads the endpoints, browses, and subscribes with publishing interval 100 ms, + sampling interval 100 ms and queue size 1, collecting notifications into a channel +- [x] `DemoScadaServerLifecycleTests`: three endpoints, `None`, `Sign` and `SignAndEncrypt`, each with one + anonymous token whose policy URI is `http://opcfoundation.org/UA/SecurityPolicy#None`; a session opens + over `None`; the namespace URI sits at index + 3; a port held by a `TcpListener` on `127.0.0.1` fails `StartAsync` with a `SeederException` naming the + port; a server stopped with a session attached starts again on the same port; a second start in the + same certificate directory leaves one certificate in `own/certs`; `Completion` is still running + before `DisposeAsync` and has completed successfully after it, `DisposeAsync` throwing nothing +- [x] run tests: `dotnet test SemiPlot/SemiPlot.Tests.Integration/SemiPlot.Tests.Integration.csproj --filter "FullyQualifiedName~DemoScada"` - + must pass before task 5 +- ⚠️ the stock stack advertises the anonymous token on `#None` at the `None` endpoint only; the `Sign` and + `SignAndEncrypt` endpoints carry it with an empty `SecurityPolicyUri` (measured 2026-10-20 by the test + client: `None #None: Anonymous:1:#None | Sign #Basic256Sha256: Anonymous:2:'' | SignAndEncrypt + #Basic256Sha256: Anonymous:2:''`), while the real server advertises `#None` at all seven. No override + was added (run limit); the tests assert one anonymous token per endpoint and `#None` at `None` only. A + session opens over `None` with the token policy taken from that endpoint's `UserIdentityTokens`. +- ➕ `StartAsync` takes no `archiveZone` yet and the server has no `VariableCount`: nothing reads either + before the address space and the loop exist, and an unused parameter fails the build (`IDE0060`). + Task 5 adds both, and `DemoScadaHost` passes the zone through. + +### Task 5: The address space and the publishing loop + +**Files:** +- Modify: `SemiPlot/SemiPlot.Tools.ArchiveSeeder/DemoScadaNodeManager.cs` +- Modify: `SemiPlot/SemiPlot.Tools.ArchiveSeeder/DemoScadaServer.cs` +- Create: `SemiPlot/SemiPlot.Tests.Integration/DemoScada/DemoScadaServerTests.cs` +- Create: `SemiPlot/SemiPlot.Tests.Integration/DemoScada/SimpleScadaNodeIds.cs` (the namespace, the folder + numbers and the node ids both test classes build; `DemoScadaHost.Options` is the one base `FollowOptions`) +- Modify: `SemiPlot/SemiPlot.Tests.Integration/DemoScada/DemoScadaServerLifecycleTests.cs` + +- [x] build the address space: `root_2`, then the group folders from `_3`, the variables, each initialised to + `ChangesAt(options, start)` with the start instant as stamp +- [x] implement the publishing loop of Technical Details +- [x] `DemoScadaServerTests` (one server per class through `IClassFixture`, 8 pens - the stand's slice, + ungrouped pen 5001 and two-group pen 1000 included - change 0.25 s, `Asia/Tokyo`): `Objects` lists + exactly one `root` of the namespace, node id `5E4D0C1A0Broot_2`, ahead of the group folders numbered + from 3; `root` organizes every pen as `5E4D0C1A0B` with its catalogue name and + `DataType` Double; each group folder organizes its primary-group pens under the same node ids; pen + 5001 sits under `root` only and no `Watchlist` folder exists; over 3 s every notification after each + pen's first has the value of `ChangesAt` at `converter.ToArchiveLocal(SourceTimestamp)`, that local + instant is a lattice instant, the stamp lies within 60 s of `DateTime.UtcNow` at receipt (a + local-as-UTC stamp under Tokyo is off by nine hours), and each pen's stamps strictly increase; each + pen gets at least 3 notifications; a + monitored item on `5E4D0C1A0B99999` fails `BadNodeIdUnknown` while the others are created +- [x] `DemoScadaServerLifecycleTests`: under `Asia/Tokyo` with change 86400 s, every first notification + carries a stamp between the instants taken before and after `StartAsync` and the value of + `ChangesAt(options, converter.ToArchiveLocal(stamp))` +- [x] run tests: `dotnet test SemiPlot/SemiPlot.Tests.Integration/SemiPlot.Tests.Integration.csproj --filter "FullyQualifiedName~DemoScada"` - + must pass before task 6 +- ➕ `SeederException` takes an optional inner exception, and the busy-port refusal carries the stack's + `BadNoCommunication` `ServiceResultException` there; `DemoScadaHost` retries only on that inner status, + so a certificate failure surfaces on the first attempt. `DemoScadaHost.ArchiveZone` fixes `Asia/Tokyo` + for every test server. The start instant is read once in `StartAsync`, ahead of the stack's start. + +### Task 6: Host the server in the follow run, with the lag + +**Files:** +- Create: `SemiPlot/SemiPlot.Tools.ArchiveSeeder/FollowEdge.cs` +- Modify: `SemiPlot/SemiPlot.Tools.ArchiveSeeder/Program.cs` +- Modify: `SemiPlot/SemiPlot.Tools.ArchiveSeeder/StaleArchiveGuard.cs` +- Modify: `SemiPlot/SemiPlot.Tests.Integration/StaleArchiveGuardTests.cs` +- Create: `SemiPlot/SemiPlot.Tests.Unit/FollowEdgeTests.cs` + +- [x] implement `FollowEdge` as in Technical Details +- [x] `FollowAsync`: the logger factory, the lagged guard read, the edge seed, the server start, the report + lines after it, the tick loop over `Advance` at the archive clock, the `Completion` check per tick, + `Advance(LocalNow())` after Ctrl+C +- [x] `StaleArchiveGuard`'s refusal names passing the previous run's `--archive-lag` beside `converge` + (`StaleArchiveGuardTests.cs:34-36` asserts only that the message contains `converge`, which it keeps; + add `.And.Contain("--archive-lag")` there, a container test run on the Linux leg) +- [x] tests: an end past the edge returns `(edge, end]`; an end at or behind the edge returns null, and the + next window still starts at the old edge; a sequence of `Advance` calls over a lagged clock, some + returning null, yields strictly increasing, non-overlapping windows whose union is `(seed, last + end]`; a final `Advance` at the unlagged clock covers the held-back span with no overlap +- [x] run tests - must pass before task 7 +- ➕ `ReportFollowPlan` now prints after the server start, so a busy port exits with its one line only; + the two new report lines keep the 16-column label alignment (`archive lag `, `opc `). + `DemoScadaServer.StartAsync` takes no Ctrl+C token: a stop pressed during the start lets it finish, then + the run writes the held-back span and stops gracefully. + +### Task 7: The stand's writer serves OPC UA + +**Files:** +- Modify: `SemiPlot/SemiPlot.AppHost/AppHost.cs` + +- [x] add `const string OpcPort = "55875";` beside the other argument constants and pass `--opc-port` to the + writer +- [x] `dotnet build SemiPlot/SemiPlot.AppHost/SemiPlot.AppHost.csproj` at 0 errors (the AppHost has no + test project; the stand check is Post-Completion) + +### Task 8: Verify acceptance criteria + +- [x] verify every Overview item and every `[decision]` above is in the code +- [x] run the Acceptance Evidence commands 1 and 2 +- [x] run the full suite: `dotnet test SemiPlot.slnx` +- [x] `dotnet format SemiPlot.slnx --verify-no-changes` and `dotnet terse` over the touched `.cs` files exit 0 +- [x] `git grep -n "Opc.Ua" -- SemiPlot/SemiPlot.UI SemiPlot/SemiPlot.Core SemiPlot/SemiPlot.DataSource.Postgres` + prints nothing (no viewer code) +- ⚠️ `EveryFirstNotificationAfterAStart_CarriesTheStartInstantAndTheLatticeValueThere` (change 86400 s under + `Asia/Tokyo`) failed if the Tokyo-midnight lattice instant, 15:00 UTC, fell between `StartAsync` and the + subscribe: a window of a few seconds a day, no production defect. Review phase 1 fixed it: the test + waits past a change due within 30 s before the start and asserts that the clock is still short of the + next change once every first notification has arrived. The stack logs the per-session + host-name line at `Error`, not `Warning`; the `Warning+` console factory still prints it, so Task 9 + words it as an error line. + +### Task 9: [Final] Update documentation + +**Files:** +- Modify: `docs/architecture/bench.md` +- Modify: `AGENTS.md` + +- [x] `bench.md` Ownership: the demo writer also owns the OPC UA demo server +- [x] `bench.md` The demo writer: the run command gains `--opc-port 55875`; a subsection "The OPC UA demo + server" states the address space, the start stamp, the publishing rule, the endpoints and the + anonymous token policy (`#None` on the `None` endpoint, as on the real server), the certificate directory, the console warnings, the required port and why it + has no default, the busy-port refusal, port 0 binding 4840, and the not-reproduced list; + `--archive-lag` with the no-rewind rule, the lagged guard read, the edge seed and the graceful-stop + write +- [x] `bench.md` The application bench: the writer's `--opc-port 55875`, why it is not 49875, that the stand + passes no lag yet, and that the seeder now carries the console provider while the AppHost's formatter + variables stay inert; the question table's live-edge row names the OPC port +- [x] `bench.md` The test bench: the OPC classes run in `DemoScadaCollection`, outside + `ArchiveDatabaseCollection`, need no container, and bind a free port because port 0 binds 4840 +- [x] `AGENTS.md`: the demo-writer paragraph gains one clause, "and serves every value over OPC UA on the + port `--opc-port` names (`docs/architecture/bench.md#the-opc-ua-demo-server`)", and its + `--follow ` mention gains `--opc-port `; the Container tests paragraph names `DemoScadaCollection` beside the socket classes that + need no runtime + +## Post-Completion + +*Items requiring manual intervention or external systems - no checkboxes, informational only* + +**Manual verification** (on this machine, with Simple-Scada's own server possibly on 49875). Steps 4 and 5 +are the only check of `FollowAsync`'s final window and the busy-port line; the unit tests cover +`FollowEdge` alone. + +1. `dotnet run --project SemiPlot/SemiPlot.AppHost`; the writer resource prints + `opc opc.tcp://127.0.0.1:55875, 8 variables` and keeps appending. +2. `netstat -ano | findstr 55875` shows one `127.0.0.1:55875 LISTENING` line and no `0.0.0.0` line; no + firewall prompt appears. +3. An `asyncua` browse of `opc.tcp://127.0.0.1:55875` (the scratch probe `opc_browse.py` walks `Objects` + and prints node id, browse name and value; `uaclient`'s `uals` serves the same) lists the five group folders and `root` under + `ns=3`, eight variables under `root` named like the sidebar's pens, values moving every 0.5 s. +4. Stop the stand; run `converge` and then the writer by hand, both with `--change-seconds 0.5`, the + writer with `--opc-port 55875 --archive-lag 60`. After more than a minute, `SELECT max(t) FROM + public.trends` trails the clock by 60 s plus at most one tick and one change interval. Ctrl+C prints + `stopped`, and the same query then returns the last lattice instant before the stop. +5. With a process already listening on 55875, the writer exits 1 with one line naming the port. + +**Remaining slices** of `docs/plans/roadmaps/20261019-opc-live-edge-roadmap.md`: + +- opc-live-source (Closes #116): `SemiPlot.DataSource.OpcUa` feeds the live edge; the poll is deleted. It + also adds `ConfigFiles/opc/opc.yaml` (port 49875), writes 55875 into the stand's copy the way `converge` + writes 55432, adds `--archive-lag` to the AppHost's writer, pins the monitored-item queue size of 1 in a + unit test (the demo does not reproduce the empty first value), keeps the configured address over the + advertised one in a unit test, reproduces a project restart by restarting the demo server on its port + and a lost transport through a loopback relay, and takes the anonymous token policy from the endpoint + it connects to (both servers advertise `#None` on the `None` endpoint, measured 2026-10-20). Its restart + test sees the start stamp only with a change interval longer than the server's start and the subscribe + together (`bench.md#the-opc-ua-demo-server`). +- opc-settings: an OPC block after the DB block in the settings window. +- live-hold-to-edge: a grey pending gap and a render-only hold of the last value to the edge. + +**Executed by exec:** + +- branch: opc-demo-server + +## Verify it yourself + +Each check names its command. A check that needs no container runs anywhere; the container tests need +Docker. None of them binds or dials 49875. + +1. The demo server's contract (Acceptance Evidence 1), no container: + `dotnet test SemiPlot/SemiPlot.Tests.Integration/SemiPlot.Tests.Integration.csproj --filter "FullyQualifiedName~DemoScada"` + passes 16 of 16 at `7b67cbd`. The endpoints, the `#None` token on the `None` endpoint, namespace index 3, + `root_2` ahead of the group folders from `_3`, the 8 pens, the true-UTC lattice stamps under + `Asia/Tokyo`, the start stamp, `BadNodeIdUnknown`, the busy-port `SeederException`, the restart on the + same port, the reused certificate (same bytes) and the faulted publishing loop each have a test there. + On `master` the `DemoScada` classes do not exist. +2. The lattice, the command line and the follow edge (Acceptance Evidence 2): + `dotnet test SemiPlot/SemiPlot.Tests.Unit/SemiPlot.Tests.Unit.csproj --filter "FullyQualifiedName~SharedLatticeTests|FullyQualifiedName~SeederCommandTests|FullyQualifiedName~LiveTailGeneratorTests|FullyQualifiedName~FollowEdgeTests"` + passes. To see the third `SharedLatticeTests` fact go red, make `LiveTailGenerator.ChangesAt` read the + next lattice index (`change + interval`): `TheServedChangeAtEveryLatticeInstantIsTheFollowWindowsChangeRow` + fails with "differs at index 0" (measured in Task 3), and passes again once restored. +3. The whole branch (Acceptance Evidence 3, local half): `dotnet build SemiPlot.slnx` gives 0 errors and + only the AppHost's two `NU1902`; `dotnet test SemiPlot/SemiPlot.Tests.Unit/SemiPlot.Tests.Unit.csproj` + passes 2326 of 2326 and `dotnet test SemiPlot/SemiPlot.Tests.Integration/SemiPlot.Tests.Integration.csproj` + 216 of 216 at `7b67cbd`. The CI half is unobserved: the branch has not been pushed. +4. No viewer code: `git grep -n "Opc.Ua" -- SemiPlot/SemiPlot.UI SemiPlot/SemiPlot.Core SemiPlot/SemiPlot.DataSource.Postgres` + prints nothing. +5. The command line, by hand, without a database: `dotnet run --project SemiPlot/SemiPlot.Tools.ArchiveSeeder -- --connection "Host=127.0.0.1" --follow 1` + exits 1 naming `--opc-port` and 49875; adding `--opc-port 0` exits 1 naming the value; a seeding run + with `--end 2026-01-02T00:00:00 --archive-lag 5` exits 1 naming `--archive-lag` (all three checked at + `7b67cbd` with `dotnet run --no-build`). On `master` the first command passes the parser and goes on to + connect. +6. The first-notification flake fixed in review has no reliable manual repro: it failed only when the + Tokyo-midnight change (15:00 UTC) landed between `StartAsync` and the subscribe. The guard is + `WaitUntilNoChangeIsNearAsync` in `DemoScada/DemoScadaServerLifecycleTests.cs` (commit `83442f0`); run + `--filter "FullyQualifiedName~EveryFirstNotificationAfterAStart"` within 30 s before 15:00 UTC and the + test waits past the change instead of failing. +7. The stand itself, the server under the live writer, the archive lag and the busy-port line: the + Post-Completion manual steps 1-5 above.