diff --git a/.agents/skills/foundatio/SKILL.md b/.agents/skills/foundatio/SKILL.md index 9fdfe9107..a7b6d4df6 100644 --- a/.agents/skills/foundatio/SKILL.md +++ b/.agents/skills/foundatio/SKILL.md @@ -2,93 +2,92 @@ name: foundatio description: > Use when working with Foundatio infrastructure abstractions for .NET -- caching, - queuing, messaging, file storage, distributed locking, or background jobs. Apply - when using ICacheClient, IQueue, IMessageBus, IFileStorage, ILockProvider, IJob, + messaging, background jobs, file storage, distributed locking, or queuing. Apply + when using ICacheClient, IMessageBus, IJobClient, IJob, IFileStorage, ILockProvider, or resilience patterns like retry and circuit breakers. Covers in-memory and - production implementations (Redis, Azure, AWS, Kafka, RabbitMQ). Use context7 - MCP to fetch current API docs and examples. + production implementations (Redis, AWS, Azure). Use context7 MCP to fetch current + API docs and examples. --- # Foundatio -Pluggable infrastructure abstractions for distributed .NET apps. Interface-first, testable, swappable between in-memory (dev/test) and production providers (Redis, Azure, AWS) with zero application code changes. +Pluggable infrastructure abstractions for distributed .NET apps. Interface-first, testable, swappable between in-memory (dev/test) and production providers (Redis, AWS, Azure) with zero application code changes. ## Documentation via context7 Use context7 MCP for complete, up-to-date API docs and examples. The main library ID covers all abstractions and implementations: ```text -query-docs(libraryId="/foundatiofx/foundatio", query="How to configure queue retry policies and dead letter handling") +query-docs(libraryId="/foundatiofx/foundatio", query="How to configure messaging retry policies and dead letter handling") ``` Query with specific questions, not single keywords. All provider docs (Redis, Azure, AWS, Kafka, etc.) are included in the main library. +## Messaging and Jobs (current API) + +- One messaging client: `IMessageBus` in `Foundatio.Messaging`. The caller's verb decides delivery -- `SendAsync` is a command processed by exactly one handler instance across the fleet (competing consumers); `PublishAsync` is an event received once per subscribing service (a scaled service's instances compete), or by every instance when the subscription sets `PerInstance`. `SendBatchAsync` / `PublishBatchAsync` batch both verbs. Per-operation options: `MessageSendOptions` / `MessagePublishOptions` (priority, `Delay`/`DeliverAt`, TTL, correlation id, headers, `Destination`/`Topic` override). +- Handlers are topology-free. Implement `IMessageHandler` and register with `.Messaging.AddHandler(o => ...)`; a hosted service (`MessageHandlerHostedService`) starts them all and each message is dispatched in its own DI scope. `IMessageBus.SubscribeAsync` is the dynamic path and returns an `IMessageSubscription` handle. +- `MessageSubscriptionOptions` declares delivery intent: `Deliveries` (`MessageDeliveries.Sent`/`Published`/`Both`, default `Both`), `Subscription` / `SubscriptionQualifier` / `PerInstance` for subscriber-group identity, `MaxConcurrency` (default 1, preserves per-handler ordering), `MaxAttempts` / `RedeliveryBackoff` / `DeadLetterWhen` (+ `DeadLetterOn()` shorthand) retry overrides, `AckMode` (`Auto` default / `Manual`), and `Key` (subscriptions sharing a key form one competing group; their backoff/dead-letter DELEGATES are compared by identity, so share delegate instances). +- Routing is central: `.Messaging.ConfigureRouting(r => r.UseDefaultQueue(...).UseDefaultTopic(...).MapQueue(...).MapTopic(...).UseServiceIdentity(...).UseSubscriptionIdentity(...).UseConvention(...))`. Precedence: operation override > exact map > interface/base-type map > `MessageRouteAttribute` > configured default > convention > kebab-cased type name. +- Routing config doubles as topology declarations (`DestinationDeclaration` with a canonical `DestinationAddress` -- `ForQueue`/`ForTopic`/`ForSubscription`). `IMessageTopology` exposes `GetDeclarations()` / `EnsureAsync()` / `ValidateAsync()`. `.Messaging.ConfigureTopology(TopologyMode.Ensure | Validate | None)` picks whether the client creates missing destinations (default), only verifies they exist (throws at startup when missing), or never touches topology; the handler host applies the mode at startup. +- The CORE owns retry/dead-lettering identically on every transport: default `RetryPolicy` is `MaxAttempts` 5 with immediate-then-10s/20s/30s backoff (+/-20% jitter); configure via `.Messaging.ConfigureRetry(p => p with { ... })`. Dead-lettered messages go to the transport's native sink or a derived `"{source}.deadletter"` destination, stamped with `message.dead_letter.*` forensics headers (`KnownHeaders.DeadLetter*`). Never configure broker-native redrive policies. +- Message settlement: `IMessageContext` / `IMessageContext` with `CompleteAsync()`, `RejectAsync(RejectOptions)` (non-terminal = retry, optionally with `RedeliveryDelay`; `Terminal = true` = dead-letter with `Reason`/`Exception`), and `RenewLockAsync()`. Auto-ack is the default. +- Transports advertise role-aware capabilities: `ITransportInfo.GetCapabilities(DestinationRole)` returns `TransportCapabilities` (e.g. the AWS transport's queue role has a native 15-minute `MaxDeliveryDelay`; its topic role has none). Delays beyond a ceiling and store-parked retries fall back to the durable runtime store (`IScheduledDispatchStore`, satisfied by any `IJobRuntimeStore`) and are drained by the job runtime pump -- never silently truncated. +- Durable jobs: implement `IJob` (`Task RunAsync(JobExecutionContext context)`). `IJobClient.EnqueueAsync()` / `EnqueueAsync(args)` (typed payloads) returns a `JobHandle`; `IJobMonitor` queries state; `IJobWorker` executes with per-run DI scopes, bounded concurrency, and supervised lease renewal. `JobExecutionContext` gives `JobId`/`Attempt`/`CancellationToken`, `GetArguments()`, `ReportProgressAsync`, `RenewLeaseAsync`, `IsCancellationRequestedAsync`; its public constructor makes a detached context for tests. +- CRON: `.Jobs.AddCronJob("0 */6 * * *", o => ...)` with `CronJobOptions` (`Scope` Global/PerNode, `Overlap`, `MisfireWindow`, `MaxRetries`, `TimeZone`, typed `Arguments`). Scheduled automatically when the runtime pump starts. Tune the pump with `.Jobs.ConfigureRuntimePump(o => ...)` (`JobRuntimePumpOptions`: `Enabled`, `PollInterval`, `BatchSize`, `MaxJobAttempts`, `WorkerConcurrency`). +- Stable wire names: `.Messaging.RegisterMessageType("name")` and `.Jobs.Register("name")` so persisted discriminators survive assembly/namespace moves; unregistered types fall back to `Type.FullName`. +- Legacy APIs still ship under the `Foundatio.Messaging.Legacy` (old `IMessageBus`/`InMemoryMessageBus`) and `Foundatio.Jobs.Legacy` (`JobBase`, `QueueJobBase`, `JobWithLockBase`, `JobRunner`) namespaces for migration. Prefer the current API in new code. + ## Core Interfaces | Interface | Purpose | In-Memory | Production | | --------- | ------- | --------- | ---------- | | `ICacheClient` | Key-value caching with TTL | `InMemoryCacheClient` | Redis, Hybrid | -| `IQueue` | FIFO message queuing | `InMemoryQueue` | Redis, SQS, Azure | -| `IMessageBus` | Pub/sub messaging | `InMemoryMessageBus` | Redis, Kafka, RabbitMQ, Azure | +| `IMessageBus` | Commands (`SendAsync`) + events (`PublishAsync`) over one client | `InMemoryMessageTransport` | Redis Streams, AWS SQS/SNS | +| `IJobClient` / `IJobMonitor` | Submit and observe durable background jobs | `InMemoryJobRuntimeStore` | `RedisJobRuntimeStore` | +| `IQueue` | Work-item queue (classic API) | `InMemoryQueue` | Redis, SQS, Azure | | `IFileStorage` | File storage abstraction | `InMemoryFileStorage` | S3, Azure Blob, Minio | | `ILockProvider` | Distributed locking | `CacheLockProvider` | Redis-backed | -| `IJob` | Background job processing | N/A | Hosted services | | `ISerializer` / `ITextSerializer` | Binary and text serialization | `SystemTextJsonSerializer` | MessagePack, JsonNet | | `IResiliencePolicy` | Retry, circuit breaker, timeout | `ResiliencePolicyBuilder` | N/A | ## DI Registration -All services are **singletons** (maintain internal state/connections). Jobs are scoped. +Use the `AddFoundatio()` fluent builder; infrastructure services register as **singletons**. Handlers and jobs resolve in their own DI scope per message/run, so they can inject scoped dependencies. ```csharp var builder = WebApplication.CreateBuilder(args); -// Quick start -- all in-memory defaults -builder.Services.AddFoundatio(); - -// Or register individually with options -builder.Services.AddSingleton(sp => - new InMemoryCacheClient(o => o.MaxItems(1000) - .LoggerFactory(sp.GetRequiredService()))); -builder.Services.AddSingleton(); -builder.Services.AddSingleton(); -builder.Services.AddSingleton>(sp => - new InMemoryQueue()); - -// Lock provider (message bus optional but enables faster lock release via pub/sub) -builder.Services.AddSingleton(sp => - new CacheLockProvider( - sp.GetRequiredService(), - sp.GetService(), - sp.GetService(), - sp.GetService())); +builder.Services.AddFoundatio() + .Caching.UseInMemory() + .Storage.UseFolder("data") + .Locking.UseCache() + .Messaging + .ConfigureRouting(r => r + .UseServiceIdentity("billing") + .MapQueue("orders") + .MapTopic("order-events", typeof(IOrderEvent))) + .ConfigureRetry(p => p with { MaxAttempts = 5 }) + .UseInMemory() + .Messaging.AddHandler() + .Jobs.UseInMemoryRuntime() + .Jobs.Register("search.rebuild"); ``` -Swap to production by changing only DI registration: +Swap to production by changing only the provider lines: ```csharp -var redis = await ConnectionMultiplexer.ConnectAsync("localhost:6379"); +builder.Services.AddFoundatio() + .Messaging.UseRedis(connectionString: "localhost:6379") // Redis Streams transport + .Jobs.UseRedis(); // Redis job runtime store -builder.Services.AddSingleton(sp => - new RedisCacheClient(o => - { - o.ConnectionMultiplexer = redis; - o.LoggerFactory = sp.GetRequiredService(); - })); -builder.Services.AddSingleton(sp => - new RedisMessageBus(o => - { - o.Subscriber = redis.GetSubscriber(); - o.LoggerFactory = sp.GetRequiredService(); - })); -builder.Services.AddSingleton>(sp => - new RedisQueue(o => - { - o.ConnectionMultiplexer = redis; - o.LoggerFactory = sp.GetRequiredService(); - })); +// or AWS (SQS queues, SNS+SQS pub/sub; point ServiceUrl at LocalStack for local dev) +builder.Services.AddFoundatio() + .Messaging.UseAws(o => o.ResourcePrefix = "myapp"); ``` +Custom providers plug in via `.Messaging.UseTransport(...)` (any `IMessageTransport`) and `.Jobs.UseRuntimeStore(...)` (any `IJobRuntimeStore`). + ## Usage Patterns ### Caching @@ -104,29 +103,39 @@ await _cache.IncrementAsync("requests:today", 1); await _cache.RemoveByPrefixAsync("user:"); ``` -### Queues +### Messaging + +The verb carries the delivery semantic; handlers never choose queue vs. topic: ```csharp -await _queue.EnqueueAsync(new OrderWorkItem { OrderId = orderId }); +// Command: exactly one handler instance across the fleet processes it. +await _bus.SendAsync(new ResizeImage(imageId)); -var entry = await _queue.DequeueAsync(TimeSpan.FromSeconds(5)); -if (entry is not null) +// Event: every subscribing service receives one copy. +await _bus.PublishAsync(new OrderSubmitted(orderId)); +``` + +```csharp +public class SendConfirmationHandler : IMessageHandler { - await ProcessAsync(entry.Value); - await entry.CompleteAsync(); // success - // or: await entry.AbandonAsync(); // retry later + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + => _email.SendConfirmationAsync(context.Message.OrderId, cancellationToken); } + +services.AddFoundatio() + .Messaging.AddHandler(o => + { + o.MaxConcurrency = 4; // default 1 preserves per-handler ordering + o.DeadLetterOn(); // retries cannot fix validation failures + }); ``` -### Messaging (Pub/Sub) +Throwing from `HandleAsync` triggers the core retry/dead-letter policy. With `AckMode.Manual`, settle explicitly: ```csharp -await _messageBus.SubscribeAsync(async (msg, ct) => -{ - await HandleOrderCreatedAsync(msg, ct); -}); - -await _messageBus.PublishAsync(new OrderCreated { OrderId = orderId }); +await context.CompleteAsync(); +await context.RejectAsync(new RejectOptions { RedeliveryDelay = TimeSpan.FromSeconds(30) }); +await context.RejectAsync(new RejectOptions { Terminal = true, Reason = "malformed" }); ``` ### File Storage @@ -142,7 +151,8 @@ await _storage.DeleteFilesAsync("reports/old-*"); ### Distributed Locks ```csharp -await using var lck = await _locker.AcquireAsync( +// TryAcquireAsync returns null when the lock is unavailable; AcquireAsync throws instead. +await using var lck = await _locker.TryAcquireAsync( "resource:order-123", timeUntilExpires: TimeSpan.FromMinutes(1)); @@ -170,146 +180,108 @@ await policy.ExecuteAsync(async ct => ## Jobs -### Standard Job +### Durable Job -```csharp -public class CleanupJob : JobBase -{ - public CleanupJob( - TimeProvider timeProvider, - IResiliencePolicyProvider resiliencePolicyProvider, - ILoggerFactory loggerFactory = null) - : base(timeProvider, resiliencePolicyProvider, loggerFactory) { } - - protected override async Task RunInternalAsync(JobContext context) - { - await CleanupOldRecordsAsync(context.CancellationToken); - return JobResult.Success; - } -} -``` - -### Job with Lock (Singleton / Leader Election) - -`JobWithLockBase` acquires a distributed lock before each run. If the lock isn't available the run is cancelled. Implements `IJobWithOptions`. +Implement `IJob`; enqueue through `IJobClient`. Arguments are typed and persisted with the job: ```csharp -[Job(Description = "Singleton maintenance", Interval = "5s")] -public class MaintenanceJob : JobWithLockBase +public class RebuildSearchIndexJob : IJob { - private readonly ILockProvider _lockProvider; - - public MaintenanceJob( - ICacheClient cache, IMessageBus messageBus, - TimeProvider timeProvider, IResiliencePolicyProvider resiliencePolicyProvider, - ILoggerFactory loggerFactory) : base(timeProvider, resiliencePolicyProvider, loggerFactory) + public async Task RunAsync(JobExecutionContext context) { - _lockProvider = new CacheLockProvider(cache, messageBus, loggerFactory); - } + var args = context.GetArguments(); - // new CancellationToken(true) = try once, skip if lock is held - protected override Task GetLockAsync(CancellationToken cancellationToken) => - _lockProvider.AcquireAsync(nameof(MaintenanceJob), TimeSpan.FromMinutes(15), - cancellationToken: new CancellationToken(true)); + await context.ReportProgressAsync(10, "starting"); + foreach (var batch in GetBatches(args.Index)) + { + if (await context.IsCancellationRequestedAsync()) + return JobResult.Cancelled; + + await IndexBatchAsync(batch, context.CancellationToken); + await context.RenewLeaseAsync(); // heartbeat for long runs + } - protected override async Task RunInternalAsync(JobContext context) - { - await DoMaintenanceAsync(context.CancellationToken); return JobResult.Success; } } + +JobHandle handle = await _jobs.EnqueueAsync( + new RebuildSearchIndexArgs { Index = "orders" }); +JobState? state = await handle.GetStateAsync(); +await handle.RequestCancellationAsync(); ``` -### Queue Processor Job +The worker gives every run its own DI scope, claims jobs with compare-and-set transitions (no double-runs), and supervises the lease: a run is cancelled when the lease is lost to another node or renewal keeps failing past the lease window. Crashed runs are reclaimed and retried until the attempt budget (`JobRuntimePumpOptions.MaxJobAttempts`, default 3) is exhausted, then dead-lettered. + +### CRON Job ```csharp -public class OrderProcessorJob : QueueJobBase -{ - public OrderProcessorJob( - IQueue queue, - TimeProvider timeProvider, - IResiliencePolicyProvider resiliencePolicyProvider, - ILoggerFactory loggerFactory = null) - : base(queue, timeProvider, resiliencePolicyProvider, loggerFactory) { } - - protected override async Task ProcessQueueEntryAsync( - QueueEntryContext context) +services.AddFoundatio() + .Jobs.UseInMemoryRuntime() + .Jobs.AddCronJob("0 2 * * *", o => { - var item = context.QueueEntry.Value; - await ProcessOrderAsync(item.OrderId, context.CancellationToken); - return JobResult.Success; - } -} + o.Scope = ScheduledJobScope.Global; // one instance per tick (default); PerNode = every instance + o.MaxRetries = 3; + o.Arguments = new ExportArgs { Format = "csv" }; + }); ``` -### Hosting Integration +Occurrences are materialized durably through the runtime store (deduplicated across nodes and misfire windows) and executed by the auto-registered `JobRuntimePumpService`. -Requires `Foundatio.Extensions.Hosting` package: +### Legacy Jobs -```csharp -builder.Services.AddJob(o => o.WaitForStartupActions()); -builder.Services.AddCronJob("0 */6 * * *"); -builder.Services.AddDistributedCronJob("0 */6 * * *"); -``` +`JobBase`, `QueueJobBase`, and `JobWithLockBase` live in `Foundatio.Jobs.Legacy` (hosted via `Foundatio.Extensions.Hosting`'s `AddJob` / `AddCronJob` / `AddDistributedCronJob`). They still work but are the previous model; prefer `IJob` + the durable runtime for new code. ## Testing -Use `Foundatio.Xunit.v3` for test logging and DI integration. Two base classes: +### Messaging: Foundatio.Testing harness -- **`TestWithLoggingBase`** -- lightweight, no DI container. `_logger` (`ILogger`) for logging; `Log` (`ILoggerFactory`) for passing to Foundatio services. -- **`TestLoggerBase`** -- full DI via `TestLoggerFixture`. Override `ConfigureServices` to register services. `Log` (`ILogger`) for logging; `TestLogger` (`ILoggerFactory`) for passing to Foundatio services. +`Foundatio.Testing` runs the real `IMessageBus` over a recording in-memory transport -- deterministic tests without sleeps, including the retry/dead-letter path: ```csharp -using Foundatio.Caching; -using Foundatio.Xunit; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Logging; -using Xunit; +services.AddFoundatio() + .Messaging.UseTestHarness() + .Messaging.AddHandler(); -public class OrderServiceTests : TestLoggerBase -{ - public OrderServiceTests(ITestOutputHelper output, TestLoggerFixture fixture) - : base(output, fixture) { } +// resolve MessagingTestHarness from the container; start hosted services, then: +await bus.PublishAsync(new OrderPlaced(42)); +await harness.WaitForIdleAsync(); // blocks until queues and in-flight handlers drain - protected override void ConfigureServices(IServiceCollection services) - { - services.AddSingleton(sp => - new InMemoryCacheClient(o => o.LoggerFactory(TestLogger))); - services.AddSingleton(); - } +Assert.Single(harness.Published()); +Assert.Single(harness.Handled()); +Assert.Empty(harness.DeadLetteredMessages); +``` - [Fact] - public async Task GetStatusAsync_WithCachedOrder_ReturnsCachedStatus() - { - // Arrange - var cache = Services.GetRequiredService(); - await cache.SetAsync("order:123", "shipped"); - Log.LogInformation("Seeded cache with order status"); +Recordings: `SentMessages` / `PublishedMessages` / `HandledMessages` / `AbandonedMessages` (retries) / `DeadLetteredMessages`, with typed accessors `Sent()`, `Published()`, `Handled()`, `Abandoned()`, `DeadLettered()`. - // Act - var status = await Services.GetRequiredService() - .GetStatusAsync("123"); +For jobs, `new JobExecutionContext(cancellationToken, arguments: myArgs)` builds a detached context to run an `IJob` directly -- progress/lease helpers no-op and `GetArguments()` returns the supplied object. - // Assert - Assert.Equal("shipped", status); - } -} -``` +### Test logging via Foundatio.Xunit.v3 + +Two base classes: + +- **`TestWithLoggingBase`** -- lightweight, no DI container. `_logger` (`ILogger`) for logging; `Log` (`ILoggerFactory`) for passing to Foundatio services. +- **`TestLoggerBase`** -- full DI via `TestLoggerFixture`. Override `ConfigureServices` to register services. `Log` (`ILogger`) for logging; `TestLogger` (`ILoggerFactory`) for passing to Foundatio services. + +### Custom providers + +Validate a custom transport or job store against the shared conformance suites in `Foundatio.TestHarness`: inherit `MessageTransportConformanceTests` (override `CreateTransport`) and `JobRuntimeStoreConformanceTests` (override `CreateStore`). Tests skip automatically for unimplemented optional interfaces or unavailable backends. ## Gotchas -- **Lock returns null**: `TryAcquireAsync` returns `null` when the lock cannot be acquired -- always guard with `is not null` before doing work. `AcquireAsync` throws `LockAcquisitionTimeoutException` instead of returning null. +- **Handlers registered per class get their own event copy**: `AddHandler` defaults the `SubscriptionQualifier` to the handler type name, so two handler classes on one event type EACH receive every published message. Set an explicit shared `Subscription` only when they should compete. +- **Shared subscription keys compare delegates by identity**: subscriptions sharing a `Key` must pass the SAME `RedeliveryBackoff`/`DeadLetterWhen` delegate instances -- a lambda recreated per subscription is rejected as a conflicting registration. +- **Do not configure broker redrive policies**: the core owns retry/dead-lettering (SQS `maxReceiveCount`, DLX, etc. would split authority and make behavior transport-specific). +- **A runtime store needs its pump**: the DI builder auto-registers `JobRuntimePumpService` with any runtime store, but in a non-hosted process (no generic host) nothing starts it -- drive `JobScheduleProcessor`/`IJobWorker` manually or nothing drains. +- **Delayed sends beyond transport ceilings need a runtime store**: e.g. > 15 min on SQS, or any delayed publish on SNS topics. Without a store the operation fails loudly rather than truncating the delay. +- **`WaitForIdleAsync` ignores store-parked work**: delayed sends/retries parked in the runtime store are not transport activity -- drain them via the job schedule processor before asserting. +- **Lock returns null**: `TryAcquireAsync` returns `null` when the lock cannot be acquired -- always guard with `is not null`. `AcquireAsync` throws `LockAcquisitionTimeoutException` instead of returning null. - **Dispose streams and locks**: `ILock` is `IAsyncDisposable` -- use `await using`. Streams from `GetFileStreamAsync` are `IDisposable` -- use `using var`. -- **Cache TTL floor**: Expiration values below 5ms are treated as already-expired and the key is silently removed. If you compute TTL dynamically (e.g., `expiresAt - now`), guard against near-zero values. -- **Cache `GetAsync` returns `CacheValue`**: Check `result.HasValue` before accessing `result.Value`. A missing key returns `HasValue = false`, not an exception. -- **Cache stampede (thundering herd)**: The cache-aside pattern (`Get` -> miss -> load -> `Set`) is vulnerable to stampedes when a popular key expires and many callers regenerate simultaneously. Use `CacheLockProvider` to serialize regeneration: acquire a lock keyed on the cache key, double-check the cache after acquiring, and only then call the backing store. See the [Cache Stampede Protection](https://foundatio.readthedocs.io/guide/caching.html#cache-stampede-protection) docs for the full pattern. -- **Queue auto-complete**: `QueueJobBase` auto-completes entries based on `JobResult` by default. Set `AutoComplete = false` only when you need manual `CompleteAsync()`/`AbandonAsync()` control. Manual `DequeueAsync` does NOT auto-complete. -- **GetQueueEntryLockAsync error handling**: If `GetQueueEntryLockAsync` returns `null`, the queue entry is abandoned. If it throws, the entry is also abandoned and a `JobResult.FromException` is returned. Use `TryAcquireAsync` (not `AcquireAsync`) in your override since the return type is `Task`. -- **JobWithLockBase vs manual locking**: Use `JobWithLockBase` when the entire run must be single-instance (leader election). Use manual `ILockProvider.AcquireAsync` inside `JobBase` for finer-grained locking within a job. -- **JobContext.RenewLockAsync**: Call in long-running jobs (both `JobBase` and `QueueJobBase`) to prevent lock expiration mid-processing. -- **Register as singletons**: All infrastructure services (`ICacheClient`, `IMessageBus`, `IQueue`, `IFileStorage`, `ILockProvider`) maintain internal state and connections -- always register as singletons. -- **CacheLockProvider + IMessageBus**: `IMessageBus` is optional but recommended. Without it, lock release falls back to polling. With it, locks are released instantly via pub/sub notification. -- **In-memory for tests**: All in-memory implementations are functionally equivalent to production providers. Swap via DI for fast, isolated unit tests with no external dependencies. +- **Cache `GetAsync` returns `CacheValue`**: check `result.HasValue` before `result.Value`. A missing key returns `HasValue = false`, not an exception. +- **Cache stampede**: serialize regeneration of hot keys with `CacheLockProvider` (lock on the cache key, double-check after acquiring). See the [Cache Stampede Protection](https://foundatio.readthedocs.io/guide/caching.html#cache-stampede-protection) docs. +- **Register as singletons**: infrastructure services (`ICacheClient`, `IMessageBus`, `IFileStorage`, `ILockProvider`) maintain internal state and connections; the `AddFoundatio()` builder does this for you. +- **In-memory for tests**: in-memory implementations are functionally equivalent to production providers and run the same conformance suites -- swap via DI for fast, isolated tests. +- **Legacy name collisions during migration**: old and new APIs coexist (`Foundatio.Messaging.Legacy.IMessageBus` vs `Foundatio.Messaging.IMessageBus`; `Foundatio.Jobs.Legacy.IJob` vs `Foundatio.Jobs.IJob`). Disambiguate with a `using` alias in files that reference both namespaces. ## NuGet Packages @@ -317,8 +289,8 @@ public class OrderServiceTests : TestLoggerBase | Package | Provides | | ------- | -------- | -| `Foundatio` | Core interfaces, in-memory implementations, resilience, `SystemTextJsonSerializer` | -| `Foundatio.Extensions.Hosting` | `AddJob`, `AddCronJob`, `AddDistributedCronJob`, startup actions, hosted services | +| `Foundatio` | Core interfaces, in-memory implementations, messaging + durable job runtime, resilience, `SystemTextJsonSerializer` | +| `Foundatio.Extensions.Hosting` | `AddJobRuntimeService`, startup actions, legacy `AddJob`/`AddCronJob`/`AddDistributedCronJob` | ### Serializers @@ -334,10 +306,10 @@ public class OrderServiceTests : TestLoggerBase | Package | Provides | | ------- | -------- | -| `Foundatio.Redis` | Redis cache, queue, messaging, locks, storage | +| `Foundatio.Redis` | `RedisStreamsMessageTransport` (messaging), `RedisJobRuntimeStore` (jobs), plus Redis cache/queue/lock/storage | +| `Foundatio.Aws` | `AwsMessageTransport` (SQS queues, SNS+SQS pub/sub), S3 storage | | `Foundatio.AzureStorage` | Azure Blob storage, Azure Storage queues | | `Foundatio.AzureServiceBus` | Azure Service Bus queues + messaging | -| `Foundatio.AWS` | SQS queues, SQS messaging, S3 storage | | `Foundatio.Kafka` | Kafka messaging | | `Foundatio.RabbitMQ` | RabbitMQ messaging | | `Foundatio.Minio` | MinIO S3-compatible storage | @@ -348,7 +320,8 @@ public class OrderServiceTests : TestLoggerBase | Package | Provides | | ------- | -------- | -| `Foundatio.TestHarness` | Shared test base classes for validating custom implementations | +| `Foundatio.Testing` | `MessagingTestHarness` + `UseTestHarness()` recording transport for deterministic messaging tests | +| `Foundatio.TestHarness` | Conformance suites (`MessageTransportConformanceTests`, `JobRuntimeStoreConformanceTests`) for custom providers | | `Foundatio.Xunit` | xUnit v2 test logging, retry attributes | | `Foundatio.Xunit.v3` | xUnit v3 test logging, retry attributes | | `Foundatio.DataProtection` | ASP.NET Core Data Protection key storage via `IFileStorage` | diff --git a/.github/workflows/build-workflow.yml b/.github/workflows/build-workflow.yml index 89a1eb450..d964ce7e5 100644 --- a/.github/workflows/build-workflow.yml +++ b/.github/workflows/build-workflow.yml @@ -122,18 +122,23 @@ jobs: - name: Publish CI Packages if: github.event_name != 'pull_request' && github.actor != 'dependabot[bot]' run: | + # CI-feed publishing is best-effort: a feed rejecting one package (e.g. GitHub Packages returns 403 when the + # package name is linked to a different repo) must not fail a build whose compile and tests passed. Each + # failure surfaces as a warning annotation instead. Release publishing to NuGet (below) stays strict. for package in $(find . -name "*.nupkg" | grep -v "minver" | grep -v "/EmptyFiles/"); do # GitHub if [ -n "${{ secrets.GITHUB_TOKEN }}" ]; then echo "${0##*/}": Pushing $package to GitHub... - dotnet nuget push $package --source https://nuget.pkg.github.com/${{ inputs.org }}/index.json --api-key ${{ secrets.GITHUB_TOKEN }} --skip-duplicate + dotnet nuget push $package --source https://nuget.pkg.github.com/${{ inputs.org }}/index.json --api-key ${{ secrets.GITHUB_TOKEN }} --skip-duplicate \ + || echo "::warning::Failed to push ${package##*/} to GitHub Packages; continuing" fi # Feedz (remove once GitHub supports anonymous access) if [ -n "${{ secrets.FEEDZ_KEY }}" ]; then echo "${0##*/}": Pushing $package to Feedz... - dotnet nuget push $package --source https://f.feedz.io/foundatio/foundatio/nuget --api-key ${{ secrets.FEEDZ_KEY }} --skip-duplicate + dotnet nuget push $package --source https://f.feedz.io/foundatio/foundatio/nuget --api-key ${{ secrets.FEEDZ_KEY }} --skip-duplicate \ + || echo "::warning::Failed to push ${package##*/} to Feedz; continuing" fi done diff --git a/Foundatio.All.slnx b/Foundatio.All.slnx index 356595f40..a6974581f 100644 --- a/Foundatio.All.slnx +++ b/Foundatio.All.slnx @@ -1,7 +1,6 @@ - @@ -17,6 +16,9 @@ + + + @@ -69,6 +71,7 @@ + diff --git a/Foundatio.slnx b/Foundatio.slnx index ebe457b12..d6d71db09 100644 --- a/Foundatio.slnx +++ b/Foundatio.slnx @@ -1,7 +1,7 @@ - + @@ -13,8 +13,14 @@ + + + + + + @@ -24,6 +30,7 @@ + diff --git a/docs/guide/messaging-jobs-redesign.md b/docs/guide/messaging-jobs-redesign.md new file mode 100644 index 000000000..98d163e15 --- /dev/null +++ b/docs/guide/messaging-jobs-redesign.md @@ -0,0 +1,224 @@ +# Messaging and Jobs Redesign + +The redesigned messaging API is one client — `IMessageBus` in `Foundatio.Messaging` — with two verbs. The caller's verb carries the delivery semantic, and handlers are registered without any topology decision: + +- `SendAsync` — a **command** / unit of work: exactly one handler instance across the fleet processes it (competing consumers). +- `PublishAsync` — an **event**: every subscribing service receives one copy (a scaled service's instances compete for it), or every instance when the subscription opts into `PerInstance`. + +```csharp +await bus.SendAsync(new ResizeImage(id)); // one handler instance, somewhere, does the work +await bus.PublishAsync(new OrderSubmitted(id)); // every subscribing service hears about it +``` + +`SendBatchAsync` and `PublishBatchAsync` batch both verbs; the non-generic `IEnumerable` overloads accept heterogeneous batches and group by resolved route. Per-operation options are `MessageSendOptions` and `MessagePublishOptions` (priority, delay/`DeliverAt`, TTL, correlation id, headers, and a `Destination`/`Topic` override as the escape hatch). + +The legacy publish/subscribe `IMessageBus` and job APIs remain shipped under the `Foundatio.Messaging.Legacy` and `Foundatio.Jobs.Legacy` namespaces while consumers migrate. + +## The core owns behavior; transports stay simple + +The division of responsibility is deliberate: **the core owns behavior, transports stay thin.** A transport is bytes in, bytes out plus a few primitives — `IMessageTransport` is `SendAsync`, `CompleteAsync`, `AbandonAsync`, and small opt-in operation interfaces (`ISupportsPull`, `ISupportsPush`, `ISupportsDeadLetter`, `ISupportsRedeliveryDelay`, `ISupportsVisibilityTimeout`, `ISupportsLockRenewal`, `ISupportsStats`, `ISupportsProvisioning`). Everything that defines *how messaging behaves* — serialization, routing, multi-type dispatch, settlement, scheduling, and especially **retry and dead-lettering** — lives in the core and is therefore identical across every transport. There is exactly one retry authority (the core), never a tug-of-war between core policy and a broker-native redrive policy. + +Every transport API takes the same canonical identity: `DestinationAddress` (`Name`, `Role` — `Queue`/`Topic`/`Subscription`/`Binding` — and, for subscriptions, the owning `Topic`; created via `ForQueue`/`ForTopic`/`ForSubscription`). `Key` is its opaque string form (`"{topic}/{name}"` for subscriptions), so the same logical destination can never be spelled two ways on the send path versus the provisioning path. + +Facts a transport advertises are **role-aware**: the core asks `ITransportInfo.GetCapabilities(DestinationRole)` and gets a `TransportCapabilities` record (`DelayedDelivery`, `MaxDeliveryDelay`, `Priority`, `Expiration`, `Ordering`, `MaxBatchSize`, `MaxMessageBytes`). Capabilities genuinely differ by role on real brokers — the AWS transport's queue role has a native 15-minute `MaxDeliveryDelay` (SQS `DelaySeconds`) while its topic role has no native delay at all. Anything not advertised is treated as unsupported: the core validates, falls back to the runtime store, or fails loudly — a broker never silently drops a requested behavior. + +## Setup + +```csharp +services.AddFoundatio() + .Messaging + .ConfigureRouting(r => r + .UseServiceIdentity("billing") + .MapQueue("orders") + .MapTopic("order-events", typeof(IOrderEvent))) + .ConfigureRetry(p => p with { MaxAttempts = 5 }) + .ConfigureTopology(TopologyMode.Ensure) + .RegisterMessageType("order.submitted") + .UseInMemory() + .Messaging.AddHandler() + .Jobs.UseInMemoryRuntime() + .Jobs.Register("search.rebuild"); +``` + +Swap providers by swapping one line: `.Messaging.UseRedis()` (Redis Streams), `.Messaging.UseAws()` (SQS/SNS), `.Jobs.UseRedis()`, or `.Messaging.UseTransport(...)` / `.Jobs.UseRuntimeStore(...)` for anything custom. Application code depends on `IMessageBus`, `IJobClient`, and `IJobMonitor`; deployment or admin code can depend on `IMessageTopology`. + +`RegisterMessageType(name)` gives a type a stable wire discriminator so payloads survive assembly/namespace moves; unregistered types fall back to `Type.FullName` (never `AssemblyQualifiedName`). `.Jobs.Register(name)` does the same for persisted job types. + +## Handlers + +Handlers are topology-free. A handler implements `IMessageHandler` and is registered declaratively; it never decides queue-vs-topic — the sender's verb does: + +```csharp +public class SendConfirmationHandler : IMessageHandler +{ + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + => _email.SendConfirmationAsync(context.Message.OrderId, cancellationToken); +} + +services.AddFoundatio() + .Messaging.AddHandler(o => + { + o.MaxConcurrency = 4; + o.DeadLetterOn(); + }); +``` + +Each message is dispatched to the handler in its own DI scope (scoped dependencies work), and a single auto-registered hosted service (`MessageHandlerHostedService`) starts every registered handler at app start and disposes them at shutdown. Throwing from `HandleAsync` triggers the core retry/dead-letter policy. Each handler class defaults to its own subscriber group (the `SubscriptionQualifier` is set to the handler type name), so every handler registered for an event type receives its own copy of each published message. + +For dynamic subscriptions, `IMessageBus.SubscribeAsync(handler, options)` returns an `IMessageSubscription` handle (`Key`, `Destination`, `Topic`, `Subscription`, `Source`); disposing it detaches the handler. + +### Subscription options + +A subscription listens on the type's two delivery channels — sent commands and published events — and `MessageSubscriptionOptions` declares its intent: + +- **`Deliveries`** — `MessageDeliveries.Sent`, `Published`, or `Both` (default). A handler that only ever consumes commands (or only events) states that so no idle listener is wired — and so a queue-only or topic-only transport can serve it. The default `Both` quietly narrows to what the transport supports; explicitly requesting a single channel the transport cannot serve throws `NotSupportedException`. +- **`Subscription`** — the subscriber-group identity for published messages. Defaults to the service identity, so all instances of a service share one subscription and compete. **`SubscriptionQualifier`** distinguishes groups within one service (`"{service-identity}.{qualifier}"`). **`PerInstance`** gives every running instance its own unique subscription (cache invalidation, config reload) and is mutually exclusive with `Subscription`. +- **`MaxConcurrency`** — messages processed concurrently per instance. Default 1: the only default that preserves per-handler ordering, and each handler already gets its own concurrent stream (10 handlers = 10 parallel consumers). Raise it for I/O-bound, order-agnostic handlers. +- **`MaxAttempts`**, **`RedeliveryBackoff`**, **`DeadLetterWhen`** — per-subscription retry overrides; null inherits the default `RetryPolicy`. **`DeadLetterOn()`** is the by-type shorthand for `DeadLetterWhen` and composes (call once per exception type). +- **`AckMode`** — `Auto` (default) or `Manual`. +- **`RouteType`**, **`Destination`**, **`Topic`** — grouped/interface consumption and per-subscription route overrides. +- **`Key`** — consumer identity. Subscriptions sharing a `Key` on the same channel form one competing group and must configure identical failure policies; the backoff/`DeadLetterWhen` **delegates are compared by identity**, so share the same delegate instances — a lambda recreated per subscription is rejected as a conflicting registration. + +Delivery semantics are never invisible: each subscription logs its effective topology (destination, subscriber group, concurrency, retry posture) once at subscribe time. + +## Routing and topology + +`IMessageRouter` resolves the queue destination and topic for a message type. The default router's precedence: + +```text +operation override > exact type map > interface/base-type map > MessageRouteAttribute > configured default > convention > kebab-cased type name +``` + +Configure routes once with `ConfigureRouting` (a `MessageRoutingOptionsBuilder`): `UseDefaultQueue`, `UseDefaultTopic`, `MapQueue` / `MapQueue(destination, params Type[])`, `MapTopic` / `MapTopic(topic, params Type[])`, `UseServiceIdentity`, `UseSubscriptionIdentity`, and `UseConvention`. `UseServiceIdentity` names the service (the default subscriber-group identity); when unset it falls back to the `FOUNDATIO_SUBSCRIPTION_ID` / `FOUNDATIO_SERVICE_ID` environment variables, then the kebab-cased app name. + +**Routing configuration is also the topology declaration source.** `UseDefaultQueue`, `UseDefaultTopic`, `MapQueue`, and `MapTopic` declare the destinations they name, and setting a service/subscription identity declares the subscription on each configured topic — as `DestinationDeclaration` values carrying the *same* canonical `DestinationAddress` the runtime later sends to and receives from, so provisioning and runtime can never disagree on identity. Per-operation overrides are deliberately excluded: they are exceptional one-off routes. + +```csharp +IMessageTopology topology = provider.GetRequiredService(); +IReadOnlyList declarations = topology.GetDeclarations(); +await topology.EnsureAsync(); // deploy/admin process with create permissions +await topology.ValidateAsync(); // check-only; throws naming what is missing +``` + +`TopologyMode` (via `ConfigureTopology`) governs how the client administers topology at runtime and at startup: + +- **`Ensure`** (default) — create missing destinations on first use, and the handler host ensures the declared topology before any handler starts consuming. +- **`Validate`** — never create; verify each destination exists and throw when missing. Startup fails at boot instead of surfacing as runtime send errors. +- **`None`** — no topology calls at all; everything is pre-provisioned out of band. + +The mode governs the core's provisioning calls; combine `Validate`/`None` with transport knobs such as `AwsMessageTransportOptions.AutoCreateDestinations = false` for a fully locked-down broker. + +## Delivery settlement + +Received messages surface as `IMessageContext` / `IMessageContext` (id, body, headers, correlation id, priority, `Attempts`) and settle with two verbs: + +```csharp +await context.CompleteAsync(); // handled successfully +await context.RejectAsync(); // retry (redelivery) +await context.RejectAsync(new RejectOptions { RedeliveryDelay = TimeSpan.FromSeconds(30) }); +await context.RejectAsync(new RejectOptions { Terminal = true, Reason = "validation", Exception = ex }); +await context.RenewLockAsync(); // long handler heartbeat +``` + +A non-terminal reject returns the message for redelivery (optionally after `RedeliveryDelay`); `Terminal = true` means "never redeliver" and routes the message to the dead-letter sink with the `Reason` and `Exception` forensics attached. `RejectOptions.BestEffortDelay` lets a delay the transport cannot honor degrade to immediate redelivery instead of failing (the core's own retry policy uses best-effort delays; an explicit caller delay defaults to strict). + +Auto-ack is the default: a handler that returns without settling is completed, and a handler that throws is rejected per the retry policy. Manual settlement is opt-in with `AckMode.Manual`. + +## Retry and dead-lettering + +The core owns retry and dead-lettering, so behavior is identical on every transport. A transport only redelivers abandoned messages and optionally exposes a dead-letter sink; the core decides how many times to retry, how long to wait, and when to give up — using the broker's own delivery count as the crash-safe attempt counter, so the core owns the *policy* without owning durable retry *state*. + +The default `RetryPolicy`: `MaxAttempts` 5, and `RetryPolicy.DefaultBackoff` — an immediate first retry, then 10s/20s/30s (capped) with ±20% jitter, the delay shape mature messaging stacks converged on. Configure the default and override per subscription: + +```csharp +services.AddFoundatio() + .Messaging.ConfigureRetry(p => p with + { + MaxAttempts = 5, + Backoff = attempt => TimeSpan.FromSeconds(Math.Min(30, Math.Pow(2, attempt))), + DeadLetterWhen = ex => ex is ValidationException, // unrecoverable: dead-letter immediately + DeadLetterDestination = "orders-dead-letter" // null derives "{source}.deadletter" + }); +``` + +Deserialization failures are always treated as unrecoverable. Where a dead-lettered message lands, in order of preference: the transport's native dead-letter sink (`ISupportsDeadLetter`), otherwise the configured or derived (`"{source}.deadletter"`) dead-letter destination written by the core. Dead-lettered messages carry forensics headers — `message.dead_letter.reason`, `.attempts`, `.exception_type`, `.exception_message`, `.exception_stack`, `.failed_at`, and `.original_destination` (`KnownHeaders.DeadLetter*`) — so a dead message is triageable with plain transport tooling, and `ISupportsDeadLetter.ReceiveDeadLetteredAsync` reads raw entries back (including poison payloads that never deserialized). + +We deliberately do **not** configure broker-native redrive policies (SQS `maxReceiveCount`, Azure Service Bus `MaxDeliveryCount`, RabbitMQ DLX): that would split authority between broker and core and make behavior transport-specific. A destination's structural creation knobs, if any, are limited to `DestinationDeclaration.ProviderArguments`. + +### Delays and the runtime-store fallback + +A send delay or redelivery backoff is served natively when the transport supports it within its advertised ceiling (`TransportCapabilities.MaxDeliveryDelay`, `ISupportsRedeliveryDelay.MaxRedeliveryDelay`). A delay the broker cannot honor — beyond SQS's 15-minute delivery delay, or any delayed publish on SNS — is parked in the durable runtime store instead of being silently truncated: `MessageBusOptions.RuntimeStore` takes an `IScheduledDispatchStore` (any `IJobRuntimeStore` satisfies it; the DI builder wires it automatically when a runtime store is configured), and the job runtime pump dispatches parked messages when due. If neither native support nor a store is available, the operation fails loudly rather than dropping the delay. + +### Unmatched message types + +A message arriving on a shared destination whose type has no registered consumer on this node — a newer message type mid rolling-deploy, or a misconfiguration — is surfaced loudly: it increments the `foundatio.messaging.unhandled` metric and throws `UnhandledMessageTypeException`, isolated to that one message so the receive loop and the other type handlers keep running. It is retried so a node that *does* handle the type can pick it up, and finally dead-lettered as `"no-handler"` after `RetryPolicy.UnmatchedMaxAttempts` (default 50) — a genuinely orphaned type cannot loop forever. + +## Jobs + +`IJobClient` submits durable work and returns a `JobHandle`; `IJobWorker` claims and executes; `IJobMonitor` queries state; `IJobRuntimeStore` persists all of it. Jobs implement `IJob`: + +```csharp +public class RebuildSearchIndexJob : IJob +{ + public async Task RunAsync(JobExecutionContext context) + { + var args = context.GetArguments(); + await context.ReportProgressAsync(50, "halfway"); + return JobResult.Success; + } +} + +JobHandle handle = await jobs.EnqueueAsync(new RebuildSearchIndexArgs { Index = "orders" }); +JobState? state = await handle.GetStateAsync(); +await handle.RequestCancellationAsync(); +``` + +**Typed payloads.** `EnqueueAsync(args)` serializes the arguments into the durable `JobState.Payload` (with `PayloadType` stored as a discriminator for forensics); the job reads them via `JobExecutionContext.GetArguments()`, guarded by `HasArguments`. Mismatches throw a descriptive exception naming the stored type. + +**Execution context.** `JobExecutionContext` carries `JobId`, `Attempt`, and the `CancellationToken`, plus the store-backed helpers useful inside job code: `ReportProgressAsync`, `RenewLeaseAsync` (heartbeat for long runs), and `IsCancellationRequestedAsync` (cooperative cancellation). Its public constructor creates a *detached* context for tests — helpers no-op, and an `arguments` object surfaces through `GetArguments` without serialization. + +**The worker.** Every run gets its own async DI scope (scoped services resolve per run, not as accidental singletons). `JobWorker` runs a bounded pool — at most `maxConcurrency` jobs in flight, a slot freeing the moment a job settles — and claims are compare-and-set guarded so concurrency cannot double-run. Lease renewal is a supervised loop, not a fire-and-forget timer: a run is cancelled when its lease is lost to another node *or* when renewal keeps failing past the lease window (the lease has lapsed on the broker's clock too, so continuing would risk double-executing side effects); the terminal state transition is ownership-guarded so a stale worker cannot overwrite the new owner's state. Stale `Processing` jobs (a worker crash mid-run) are reclaimed and re-queued while attempts remain, then dead-lettered. + +### CRON scheduling + +```csharp +services.AddFoundatio() + .Jobs.UseInMemoryRuntime() + .Jobs.AddCronJob("0 2 * * *", o => + { + o.MaxRetries = 3; + o.Arguments = new ExportArgs { Format = "csv" }; + }); +``` + +`AddCronJob(cron, o => ...)` registers a `ScheduledJobDefinition`; `CronJobOptions` covers `Name`, `Scope` (`Global` = one instance per tick, `PerNode` = every instance), `Overlap` (`SkipIfRunning` default), `MisfireWindow`, `MaxRetries`, `TimeZone`, `Enabled`, and typed `Arguments` serialized into every occurrence's payload. Definitions are scheduled automatically when the pump starts — no manual `IJobScheduler.ScheduleAsync` call. The scheduler materializes every occurrence due within the misfire window (not just the latest) as durable, deduplicated store entries, and owns occurrence recovery with its own per-definition retry/dead-letter budget. + +### The runtime pump + +`JobRuntimePumpService` is registered automatically with any runtime store, so a configured store can never silently accumulate work that nothing drains. Each poll it materializes CRON occurrences, then runs an **overlapped execution pass** — dispatching due work (message dispatches before job occurrences, so the messaging delayed-delivery fallback is never head-of-line blocked by a long job), recovering stale jobs, and running queued jobs. Scheduling keeps its cadence even while a long pass runs. Tune with `ConfigureRuntimePump`: `JobRuntimePumpOptions.Enabled` (false = manual control), `PollInterval` (1s), `BatchSize` (100), `MaxJobAttempts` (3), and `WorkerConcurrency` (1; every in-flight job still gets its own DI scope, lease, and cancellation watcher). + +## Testing + +`Foundatio.Testing` runs the real bus over a recording in-memory transport for deterministic, sleep-free tests: + +```csharp +services.AddFoundatio() + .Messaging.UseTestHarness() + .Messaging.AddHandler(); + +// start hosted services, then: +await bus.PublishAsync(new OrderPlaced(42)); +await harness.WaitForIdleAsync(); +Assert.Single(harness.Published()); +Assert.Empty(harness.DeadLetteredMessages); +``` + +Resolve `MessagingTestHarness` from the container. `WaitForIdleAsync` blocks until every destination has nothing queued and nothing in flight (throws a `TimeoutException` naming the still-busy destinations). Recordings cover every movement — `SentMessages`, `PublishedMessages`, `HandledMessages`, `AbandonedMessages`, `DeadLetteredMessages`, with typed accessors `Sent()` / `Published()` / `Handled()` / `Abandoned()` / `DeadLettered()` — so the core retry/dead-letter path is directly assertable: a message redelivered N times and then dead-lettered shows up as N abandonments plus one dead-letter. + +## Providers + +- **In-memory** (`InMemoryMessageTransport`, `InMemoryJobRuntimeStore`) — the reference implementation for local dev and tests; supports every operation interface. +- **Redis** (`Foundatio.Redis`) — `RedisStreamsMessageTransport` (FIFO streams; delays route through the runtime store) and `RedisJobRuntimeStore`, wired via `.Messaging.UseRedis()` / `.Jobs.UseRedis()` over one shared connection. +- **AWS** (`Foundatio.Aws`) — `AwsMessageTransport` (queues on SQS, pub/sub on SNS+SQS) via `.Messaging.UseAws()`; role-aware capabilities as above, `AutoCreateDestinations` to control implicit resource creation, and LocalStack support via `ServiceUrl`. + +A new provider is validated against the shared conformance suites in `Foundatio.TestHarness`: `MessageTransportConformanceTests` (send/receive, settlement, redelivery, dead-letter, visibility, provisioning — tests skip per unimplemented operation interface) and `JobRuntimeStoreConformanceTests` (state round-trips, CAS transitions, leases, stale recovery including the renew-during-reclaim race, and scheduled-dispatch claiming, driven by a fake time provider). diff --git a/samples/Foundatio.AppHost/Foundatio.AppHost.csproj b/samples/Foundatio.AppHost/Foundatio.AppHost.csproj index 0fd055124..7dcbfef32 100644 --- a/samples/Foundatio.AppHost/Foundatio.AppHost.csproj +++ b/samples/Foundatio.AppHost/Foundatio.AppHost.csproj @@ -14,7 +14,7 @@ - + diff --git a/samples/Foundatio.AppHost/Program.cs b/samples/Foundatio.AppHost/Program.cs index e5d8cc8ff..93ebe54aa 100644 --- a/samples/Foundatio.AppHost/Program.cs +++ b/samples/Foundatio.AppHost/Program.cs @@ -11,17 +11,26 @@ .WithRedisInsight(b => b.WithEndpointProxySupport(false).WithContainerName("Foundatio-RedisInsight") .WithUrlForEndpoint("http", u => u.DisplayText = "Cache")); -builder.AddProject("Foundatio-HostingSample") +// LocalStack provides AWS SQS/SNS locally so the messaging sample's AWS transport works with no cloud account. +var localstack = builder.AddContainer("localstack", "localstack/localstack", "3") + .WithContainerName("Foundatio-LocalStack") + .WithEnvironment("SERVICES", "sqs,sns") + .WithEndpoint(port: 4566, targetPort: 4566, scheme: "http", name: "gateway") + .WithHttpHealthCheck("/_localstack/health", endpointName: "gateway"); + +// The redesigned messaging + durable-jobs sample, scaled to 3 replicas so you can watch the queue load-balance across +// instances, the pub/sub topic fan out to every instance, and durable/CRON jobs get claimed by a single instance. +// Messaging runs on AWS (SQS/SNS via LocalStack) and durable jobs on Redis. WithReference(cache) supplies the "Redis" +// connection string UseRedis() reads; the Aws__* settings point UseAws() at LocalStack (which accepts any credentials). +builder.AddProject("Foundatio-MessagingSample") .WithExternalHttpEndpoints() .WithReplicas(3) .WithReference(cache) .WaitFor(cache) - .WithArgs("all") - .WithUrls(u => - { - u.Urls.Clear(); - u.Urls.Add(new ResourceUrlAnnotation { Url = "/jobs/status", DisplayText = "Job Status", Endpoint = u.GetEndpoint("http") }); - u.Urls.Add(new ResourceUrlAnnotation { Url = "/jobs/run", DisplayText = "Run Job", Endpoint = u.GetEndpoint("http") }); - }); + .WaitFor(localstack) + .WithEnvironment("Aws__ServiceUrl", localstack.GetEndpoint("gateway")) + .WithEnvironment("Aws__AccessKey", "test") + .WithEnvironment("Aws__SecretKey", "test") + .WithEnvironment("Aws__ResourcePrefix", "fnd-sample-"); await builder.Build().RunAsync(); diff --git a/samples/Foundatio.HostingSample/Foundatio.HostingSample.csproj b/samples/Foundatio.HostingSample/Foundatio.HostingSample.csproj deleted file mode 100644 index f4dfa2769..000000000 --- a/samples/Foundatio.HostingSample/Foundatio.HostingSample.csproj +++ /dev/null @@ -1,24 +0,0 @@ - - - - net10.0 - False - false - REDIS - - - - - - - - - - - - - - - - - diff --git a/samples/Foundatio.HostingSample/Jobs/EveryMinuteJob.cs b/samples/Foundatio.HostingSample/Jobs/EveryMinuteJob.cs deleted file mode 100644 index e879b104a..000000000 --- a/samples/Foundatio.HostingSample/Jobs/EveryMinuteJob.cs +++ /dev/null @@ -1,33 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Caching; -using Foundatio.Jobs; -using Microsoft.Extensions.Logging; - -namespace Foundatio.HostingSample; - -public class EveryMinuteJob : IJob -{ - private readonly ICacheClient _cacheClient; - private readonly ILogger _logger; - - public EveryMinuteJob(ILoggerFactory loggerFactory, ICacheClient cacheClient) - { - _cacheClient = cacheClient; - _logger = loggerFactory.CreateLogger(); - } - - public async Task RunAsync(CancellationToken cancellationToken = default) - { - var runCount = await _cacheClient.IncrementAsync("EveryMinuteJob"); - - _logger.LogInformation("EveryMinuteJob Run Count={Count} Thread={ManagedThreadId}", runCount, Thread.CurrentThread.ManagedThreadId); - - await Task.Delay(TimeSpan.FromSeconds(30)); - - _logger.LogInformation("EveryMinuteJob Complete"); - - return JobResult.Success; - } -} diff --git a/samples/Foundatio.HostingSample/Jobs/Sample1Job.cs b/samples/Foundatio.HostingSample/Jobs/Sample1Job.cs deleted file mode 100644 index bea203a60..000000000 --- a/samples/Foundatio.HostingSample/Jobs/Sample1Job.cs +++ /dev/null @@ -1,42 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Jobs; -using Foundatio.Resilience; -using Microsoft.Extensions.Logging; - -namespace Foundatio.HostingSample; - -[Job(Description = "Sample 1 job", Interval = "5s", IterationLimit = 5)] -public class Sample1Job : IJob -{ - private readonly IResiliencePolicy _policy; - private readonly ILogger _logger; - private int _iterationCount = 0; - - public Sample1Job(IResiliencePolicyProvider provider, ILoggerFactory loggerFactory) - { - // get policy for Sample1Job and if not found, try to get policy for IJob, then fallback to default policy - _policy = provider.GetPolicy(); - _logger = loggerFactory.CreateLogger(); - } - - public async Task RunAsync(CancellationToken cancellationToken = default) - { - return await _policy.ExecuteAsync(async _ => - { - int count = Interlocked.Increment(ref _iterationCount); - _logger.LogTrace("Sample1Job Run #{IterationCount} Thread={ManagedThreadId}", _iterationCount, Thread.CurrentThread.ManagedThreadId); - - if (count < 3) - { - _logger.LogInformation("Sample1Job Run #{IterationCount} Thread={ManagedThreadId} - Simulating failure", _iterationCount, Thread.CurrentThread.ManagedThreadId); - throw new InvalidOperationException("Simulated failure"); - } - - await Task.Delay(5000, cancellationToken); - - return JobResult.Success; - }, cancellationToken); - } -} diff --git a/samples/Foundatio.HostingSample/Jobs/Sample2Job.cs b/samples/Foundatio.HostingSample/Jobs/Sample2Job.cs deleted file mode 100644 index dc7fbdccd..000000000 --- a/samples/Foundatio.HostingSample/Jobs/Sample2Job.cs +++ /dev/null @@ -1,41 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Jobs; -using Microsoft.Extensions.Diagnostics.HealthChecks; -using Microsoft.Extensions.Logging; - -namespace Foundatio.HostingSample; - -[Job(Description = "Sample 2 job", Interval = "15s", IterationLimit = 24)] -public class Sample2Job : IJob, IHealthCheck -{ - private readonly ILogger _logger; - private int _iterationCount = 0; - private DateTime? _lastRun = null; - - public Sample2Job(ILoggerFactory loggerFactory) - { - _logger = loggerFactory.CreateLogger(); - } - - public Task RunAsync(CancellationToken cancellationToken = default) - { - _lastRun = DateTime.UtcNow; - Interlocked.Increment(ref _iterationCount); - _logger.LogTrace("Sample2Job Run #{IterationCount} Thread={ManagedThreadId}", _iterationCount, Thread.CurrentThread.ManagedThreadId); - - return Task.FromResult(JobResult.Success); - } - - public Task CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default) - { - if (!_lastRun.HasValue) - return Task.FromResult(HealthCheckResult.Healthy("Job has not been run yet.")); - - if (DateTime.UtcNow.Subtract(_lastRun.Value) > TimeSpan.FromSeconds(5)) - return Task.FromResult(HealthCheckResult.Unhealthy("Job has not run in the last 5 seconds.")); - - return Task.FromResult(HealthCheckResult.Healthy("Job has run in the last 5 seconds.")); - } -} diff --git a/samples/Foundatio.HostingSample/Jobs/SampleLockJob.cs b/samples/Foundatio.HostingSample/Jobs/SampleLockJob.cs deleted file mode 100644 index 9cae2fc83..000000000 --- a/samples/Foundatio.HostingSample/Jobs/SampleLockJob.cs +++ /dev/null @@ -1,32 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Caching; -using Foundatio.Jobs; -using Foundatio.Lock; -using Microsoft.Extensions.Logging; - -namespace Foundatio.HostingSample; - -[Job(Description = "Sample lock job", Interval = "5s")] -public class SampleLockJob : JobWithLockBase -{ - private readonly ILockProvider _lockProvider; - - public SampleLockJob(ICacheClient cache, ILoggerFactory loggerFactory) : base(loggerFactory) - { - _lockProvider = new ThrottlingLockProvider(cache, 1, TimeSpan.FromMinutes(1), _timeProvider, _resiliencePolicyProvider, loggerFactory); - } - - protected override Task GetLockAsync(CancellationToken cancellationToken = default) - { - return _lockProvider.AcquireAsync(nameof(SampleLockJob), TimeSpan.FromMinutes(15), cancellationToken); - } - - protected override Task RunInternalAsync(JobContext context) - { - _logger.LogTrace("SampleLockJob Run Thread={ManagedThreadId}", Thread.CurrentThread.ManagedThreadId); - - return Task.FromResult(JobResult.Success); - } -} diff --git a/samples/Foundatio.HostingSample/MyCriticalHealthCheck.cs b/samples/Foundatio.HostingSample/MyCriticalHealthCheck.cs deleted file mode 100644 index 7d3d7ac84..000000000 --- a/samples/Foundatio.HostingSample/MyCriticalHealthCheck.cs +++ /dev/null @@ -1,18 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Microsoft.Extensions.Diagnostics.HealthChecks; - -namespace Foundatio.HostingSample; - -public class MyCriticalHealthCheck : IHealthCheck -{ - private static DateTime _startTime = DateTime.Now; - - public Task CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = new CancellationToken()) - { - return DateTime.Now.Subtract(_startTime) > TimeSpan.FromSeconds(3) ? - Task.FromResult(HealthCheckResult.Healthy("Critical resource is available.")) - : Task.FromResult(HealthCheckResult.Unhealthy("Critical resource not available.")); - } -} diff --git a/samples/Foundatio.HostingSample/Program.cs b/samples/Foundatio.HostingSample/Program.cs deleted file mode 100644 index 8560a9786..000000000 --- a/samples/Foundatio.HostingSample/Program.cs +++ /dev/null @@ -1,222 +0,0 @@ -using System; -using System.Linq; -using System.Threading; -using System.Threading.Tasks; -using Foundatio; -using Foundatio.Extensions.Hosting.Jobs; -using Foundatio.Extensions.Hosting.Startup; -using Foundatio.HostingSample; -using Foundatio.Resilience; -using Foundatio.Serializer; -using Microsoft.AspNetCore.Builder; -using Microsoft.AspNetCore.Http; -using Microsoft.AspNetCore.Http.Json; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Hosting; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Options; -#if REDIS -using Microsoft.Extensions.Configuration; -using Foundatio.Redis; -using StackExchange.Redis; -#endif - -bool all = args.Contains("all", StringComparer.OrdinalIgnoreCase); -bool sample1 = all || args.Contains("sample1", StringComparer.OrdinalIgnoreCase); -bool sample2 = all || args.Contains("sample2", StringComparer.OrdinalIgnoreCase); -bool everyMinute = all || args.Contains("everyMinute", StringComparer.OrdinalIgnoreCase); -bool evenMinutes = all || args.Contains("evenMinutes", StringComparer.OrdinalIgnoreCase); - -var builder = WebApplication.CreateBuilder(args); - -// configure Foundatio services -builder.Services.AddFoundatio() - .Storage.UseFolder() - .Caching.UseInMemory() - .Locking.UseCache() - .Messaging.UseInMemory() - .AddSerializer(sp => new SystemTextJsonSerializer(sp.GetRequiredService>().Value.SerializerOptions)) - .AddResilience(b => b.WithPolicy(p => p.WithMaxAttempts(5).WithLinearDelay().WithJitter())); - -ConfigureServices(); - -// shutdown the host if no jobs are running, cron jobs are not considered running jobs -builder.Services.AddJobLifetimeService(); - -// inserts a startup action that does not complete until the critical health checks are healthy -// gets inserted as 1st startup action so that any other startup actions don't run until the critical resources are available -builder.Services.AddStartupActionToWaitForHealthChecks("Critical"); - -builder.Services.AddHealthChecks().AddCheck("My Critical Resource", tags: ["Critical"]); - -// add health check that does not return healthy until the startup actions have completed -// useful for readiness checks -builder.Services.AddHealthChecks().AddCheckForStartupActions("Critical"); - -// this gets added automatically by any AddJob call, but we might not be running any jobs, and we need it for doing dynamic jobs -builder.Services.AddJobScheduler(); - -if (everyMinute) - builder.Services.AddDistributedCronJob("* * * * *"); - -builder.Services.AddCronJob(b => b.Name("Tokyo").CronSchedule("44 4 * * *").CronTimeZone("Asia/Tokyo").JobAction(async sp => -{ - var logger = sp.GetRequiredService>(); - logger.LogInformation("Tokyo 4:44am Run Thread={ManagedThreadId}", Thread.CurrentThread.ManagedThreadId); - await Task.Delay(TimeSpan.FromSeconds(5)); -})); - -if (evenMinutes) - builder.Services.AddCronJob("EvenMinutes", "*/2 * * * *", async sp => - { - var logger = sp.GetRequiredService>(); - logger.LogInformation("EvenMinuteJob Run Thread={ManagedThreadId}", Thread.CurrentThread.ManagedThreadId); - await Task.Delay(TimeSpan.FromSeconds(30)); - logger.LogInformation("EvenMinuteJob Complete"); - }); - -if (sample1) - builder.Services.AddJob("Sample1", sp => new Sample1Job(sp.GetService(), sp.GetService()), o => o.ApplyDefaults().WaitForStartupActions().InitialDelay(TimeSpan.FromSeconds(4))); - -builder.Services.AddJob(o => o.WaitForStartupActions()); - -if (sample2) -{ - builder.Services.AddHealthChecks().AddCheck("Sample2Job"); - builder.Services.AddJob(o => o.WaitForStartupActions()); -} - -// if you don't specify priority, actions will automatically be assigned an incrementing priority starting at 0 -builder.Services.AddStartupAction("Test1", async sp => -{ - var logger = sp.GetRequiredService>(); - logger.LogTrace("Running startup 1 action"); - for (int i = 0; i < 3; i++) - { - await Task.Delay(100); - logger.LogTrace("Running startup 1 action..."); - } - - logger.LogTrace("Done running startup 1 action"); -}); - -// then these startup actions will run concurrently since they both have the same priority -builder.Services.AddStartupAction(priority: 100); -builder.Services.AddStartupAction(priority: 100); - -/*builder.Services.AddStartupAction("Test2", async sp => -{ - var logger = sp.GetRequiredService>(); - logger.LogTrace("Running startup 2 action"); - for (int i = 0; i < 2; i++) - { - await Task.Delay(50); - logger.LogTrace("Running startup 2 action..."); - } - //throw new ApplicationException("Boom goes the startup"); - logger.LogTrace("Done running startup 2 action"); -});*/ - -//s.AddStartupAction("Boom", () => throw new ApplicationException("Boom goes the startup")); - -var app = builder.Build(); - -app.MapGet("/", () => "Foundatio!"); - -app.MapGet("/jobs/status", (IJobManager jobManager, string name = null, bool? running = null, bool history = true) => - { - if (!String.IsNullOrEmpty(name)) - return Results.Ok(jobManager.GetJobStatus(name, includeHistory: history)); - - if (running.HasValue && running.Value) - return Results.Ok(jobManager.GetJobStatus(true, includeHistory: history)); - - return Results.Ok(jobManager.GetJobStatus(includeHistory: history)); - }); - -app.MapGet("/jobs/run", async (IJobManager jobManager, string name) => - { - if (String.IsNullOrWhiteSpace(name)) - return Results.BadRequest("Job name is required."); - - await jobManager.RunJobAsync(name); - - return Results.Accepted($"Job {name} started successfully."); - }); - -app.MapGet("/jobs/enable", (IJobManager jobManager, string name) => - { - if (String.IsNullOrWhiteSpace(name)) - return Results.BadRequest("Job name is required."); - - jobManager.Update(name, c => c.Enabled()); - - return Results.Ok($"Job {name} enabled successfully."); - }); - -app.MapGet("/jobs/disable", (IJobManager jobManager, string name) => - { - if (String.IsNullOrWhiteSpace(name)) - return Results.BadRequest("Job name is required."); - - jobManager.Update(name, c => c.Disabled()); - - return Results.Ok($"Job {name} disabled successfully."); - }); - -app.MapGet("/jobs/schedule", (IJobManager jobManager, string name, string cron) => -{ - if (String.IsNullOrWhiteSpace(name)) - return Results.BadRequest("Job name is required."); - - jobManager.Update(name, c => c.CronSchedule(cron)); - - return Results.Ok($"Job {name} updated successfully."); -}); - -app.MapGet("/jobs/release", async (IJobManager jobManager, string name) => -{ - if (String.IsNullOrWhiteSpace(name)) - return Results.BadRequest("Job name is required."); - - await jobManager.ReleaseLockAsync(name); - - return Results.Ok($"Job {name} lock released successfully."); -}); - -app.UseHealthChecks("/health"); -app.UseReadyHealthChecks("Critical"); - -// this middleware will return Service Unavailable until the startup actions have completed -app.UseWaitForStartupActionsBeforeServingRequests(); - -// add mvc or other request middleware after the UseWaitForStartupActionsBeforeServingRequests call - -app.Run(); - -void ConfigureServices() -{ - builder.Services.AddLogging(opt => - { - opt.AddSimpleConsole(c => c.TimestampFormat = "[HH:mm:ss] "); - }); - - builder.AddServiceDefaults(); - builder.Services.ConfigureHttpJsonOptions(o => { o.SerializerOptions.WriteIndented = true; }); - -#if REDIS - builder.Services.AddSingleton(sp => - { - var connectionString = builder.Configuration.GetConnectionString("Redis")!; - connectionString += ",abortConnect=false"; - return ConnectionMultiplexer.Connect(connectionString); - // enable redis logging - //return ConnectionMultiplexer.Connect(connectionString, o => o.LoggerFactory = sp.GetRequiredService()); - }); - - // distributed cache and messaging using redis (replaces in memory cache) - builder.Services.AddFoundatio() - .Caching.UseRedis() - .Messaging.UseRedis(); -#endif -} diff --git a/samples/Foundatio.HostingSample/Properties/launchSettings.json b/samples/Foundatio.HostingSample/Properties/launchSettings.json deleted file mode 100644 index f520048a1..000000000 --- a/samples/Foundatio.HostingSample/Properties/launchSettings.json +++ /dev/null @@ -1,27 +0,0 @@ -{ - "$schema": "https://json.schemastore.org/launchsettings.json", - "profiles": { - "http": { - "commandName": "Project", - "commandLineArgs": "all", - "dotnetRunMessages": true, - "launchBrowser": false, - "launchUrl": "jobstatus", - "applicationUrl": "http://localhost:5324", - "environmentVariables": { - "ASPNETCORE_ENVIRONMENT": "Development" - } - }, - "https": { - "commandName": "Project", - "commandLineArgs": "all", - "dotnetRunMessages": true, - "launchBrowser": false, - "launchUrl": "jobstatus", - "applicationUrl": "https://localhost:7580;http://localhost:5324", - "environmentVariables": { - "ASPNETCORE_ENVIRONMENT": "Development" - } - } - } -} diff --git a/samples/Foundatio.HostingSample/ServiceDefaults.cs b/samples/Foundatio.HostingSample/ServiceDefaults.cs deleted file mode 100644 index fbd3284d7..000000000 --- a/samples/Foundatio.HostingSample/ServiceDefaults.cs +++ /dev/null @@ -1,57 +0,0 @@ -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Logging; -using OpenTelemetry; -using OpenTelemetry.Metrics; -using OpenTelemetry.Trace; - -namespace Microsoft.Extensions.Hosting; - -public static class Extensions -{ - public static TBuilder AddServiceDefaults(this TBuilder builder) where TBuilder : IHostApplicationBuilder - { - builder.ConfigureOpenTelemetry(); - - return builder; - } - - public static TBuilder ConfigureOpenTelemetry(this TBuilder builder) where TBuilder : IHostApplicationBuilder - { - builder.Logging.AddOpenTelemetry(logging => - { - logging.IncludeFormattedMessage = true; - logging.IncludeScopes = true; - }); - - builder.Services.AddOpenTelemetry() - .WithMetrics(metrics => - { - metrics.AddAspNetCoreInstrumentation() - .AddHttpClientInstrumentation() - .AddRuntimeInstrumentation() - .AddMeter("Foundatio"); - }) - .WithTracing(tracing => - { - tracing.AddAspNetCoreInstrumentation() - .AddHttpClientInstrumentation() - .AddSource("Foundatio"); - }); - - builder.AddOpenTelemetryExporters(); - - return builder; - } - - private static TBuilder AddOpenTelemetryExporters(this TBuilder builder) where TBuilder : IHostApplicationBuilder - { - var useOtlpExporter = !string.IsNullOrWhiteSpace(builder.Configuration["OTEL_EXPORTER_OTLP_ENDPOINT"]); - - if (useOtlpExporter) - { - builder.Services.AddOpenTelemetry().UseOtlpExporter(); - } - - return builder; - } -} diff --git a/samples/Foundatio.HostingSample/Startup/MyStartupAction.cs b/samples/Foundatio.HostingSample/Startup/MyStartupAction.cs deleted file mode 100644 index 8a87e059d..000000000 --- a/samples/Foundatio.HostingSample/Startup/MyStartupAction.cs +++ /dev/null @@ -1,42 +0,0 @@ -using System; -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Caching; -using Foundatio.Extensions.Hosting.Jobs; -using Foundatio.Extensions.Hosting.Startup; -using Microsoft.Extensions.Logging; - -namespace Foundatio.HostingSample; - -public class MyStartupAction : IStartupAction -{ - private readonly IJobManager _jobManager; - private readonly ICacheClient _cacheClient; - private readonly ILogger _logger; - - public MyStartupAction(IJobManager jobManager, ICacheClient cacheClient, ILogger logger) - { - _jobManager = jobManager; - _cacheClient = cacheClient; - _logger = logger; - } - - public async Task RunAsync(CancellationToken cancellationToken = default) - { - // set next run to be far in the past so it runs immediately - await _cacheClient.SetAsync("jobs:every_minute:nextrun", DateTime.UtcNow.AddDays(-1)); - - for (int i = 0; i < 5; i++) - { - _logger.LogTrace("MyStartupAction Run Thread={ManagedThreadId}", Thread.CurrentThread.ManagedThreadId); - await Task.Delay(500); - } - - _jobManager.AddOrUpdate("MyJob", j => j.CronSchedule("* * * * *").JobAction(async () => - { - _logger.LogInformation("Running MyJob"); - await Task.Delay(1000); - _logger.LogInformation("MyJob Complete"); - })); - } -} diff --git a/samples/Foundatio.HostingSample/Startup/OtherStartupAction.cs b/samples/Foundatio.HostingSample/Startup/OtherStartupAction.cs deleted file mode 100644 index 96cce9a33..000000000 --- a/samples/Foundatio.HostingSample/Startup/OtherStartupAction.cs +++ /dev/null @@ -1,25 +0,0 @@ -using System.Threading; -using System.Threading.Tasks; -using Foundatio.Extensions.Hosting.Startup; -using Microsoft.Extensions.Logging; - -namespace Foundatio.HostingSample; - -public class OtherStartupAction : IStartupAction -{ - private readonly ILogger _logger; - - public OtherStartupAction(ILogger logger) - { - _logger = logger; - } - - public async Task RunAsync(CancellationToken cancellationToken = default) - { - for (int i = 0; i < 5; i++) - { - _logger.LogTrace("OtherStartupAction Run Thread={ManagedThreadId}", Thread.CurrentThread.ManagedThreadId); - await Task.Delay(900); - } - } -} diff --git a/samples/Foundatio.HostingSample/appsettings.Development.json b/samples/Foundatio.HostingSample/appsettings.Development.json deleted file mode 100644 index a34cd70c5..000000000 --- a/samples/Foundatio.HostingSample/appsettings.Development.json +++ /dev/null @@ -1,8 +0,0 @@ -{ - "Logging": { - "LogLevel": { - "Default": "Information", - "Microsoft.AspNetCore": "Warning" - } - } -} diff --git a/samples/Foundatio.HostingSample/appsettings.json b/samples/Foundatio.HostingSample/appsettings.json deleted file mode 100644 index 49bf8624f..000000000 --- a/samples/Foundatio.HostingSample/appsettings.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "Logging": { - "LogLevel": { - "Default": "Information", - "Microsoft.AspNetCore": "Warning", - "Foundatio": "Information" - } - }, - "AllowedHosts": "*" -} diff --git a/samples/Foundatio.MessagingSample/Foundatio.MessagingSample.csproj b/samples/Foundatio.MessagingSample/Foundatio.MessagingSample.csproj new file mode 100644 index 000000000..f532dbd33 --- /dev/null +++ b/samples/Foundatio.MessagingSample/Foundatio.MessagingSample.csproj @@ -0,0 +1,16 @@ + + + + net10.0 + enable + enable + false + + + + + + + + + diff --git a/samples/Foundatio.MessagingSample/Handlers.cs b/samples/Foundatio.MessagingSample/Handlers.cs new file mode 100644 index 000000000..563aa7dbf --- /dev/null +++ b/samples/Foundatio.MessagingSample/Handlers.cs @@ -0,0 +1,33 @@ +using Foundatio.Messaging; + +namespace Foundatio.MessagingSample; + +/// A short per-process id so you can see which instance handled each message/job when scaled to replicas. +public sealed record InstanceInfo(string Id); + +/// +/// Handles orders. Registration carries no topology — orders arrive here because the endpoint calls +/// bus.SendAsync, so exactly one running instance processes each order (competing consumers). Resolved from DI +/// per message; throwing would trigger retry/dead-letter. +/// +public sealed class ProcessOrderHandler(InstanceInfo instance, ILogger logger) : IMessageHandler +{ + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + { + logger.LogInformation("[{Instance}] processed order: {Quantity} x {Product}", instance.Id, context.Message.Quantity, context.Message.Product); + return Task.CompletedTask; + } +} + +/// +/// Handles announcements published via bus.PublishAsync. Registered with PerInstance = true, so every +/// running replica receives its own copy — without it, the default is once per service (replicas compete). +/// +public sealed class AnnouncementHandler(InstanceInfo instance, ILogger logger) : IMessageHandler +{ + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + { + logger.LogInformation("[{Instance}] announcement: {Text}", instance.Id, context.Message.Text); + return Task.CompletedTask; + } +} diff --git a/samples/Foundatio.MessagingSample/Jobs.cs b/samples/Foundatio.MessagingSample/Jobs.cs new file mode 100644 index 000000000..6ec9c7c04 --- /dev/null +++ b/samples/Foundatio.MessagingSample/Jobs.cs @@ -0,0 +1,59 @@ +using Foundatio.Jobs; + +namespace Foundatio.MessagingSample; + +/// +/// A durable, on-demand job (submitted via POST /reports). It runs on whichever instance's runtime pump claims +/// it, and reports progress through its so GET /reports/{id} can observe it. +/// +public sealed class GenerateReportJob(InstanceInfo instance, ILogger logger) : IJob +{ + public async Task RunAsync(JobExecutionContext context) + { + logger.LogInformation("[{Instance}] generating report {JobId}", instance.Id, context.JobId); + + for (int percent = 25; percent <= 100; percent += 25) + { + await Task.Delay(TimeSpan.FromMilliseconds(250), context.CancellationToken); + await context.ReportProgressAsync(percent, $"{percent}% complete", context.CancellationToken); + } + + return JobResult.Success; + } +} + +// The recurring (CRON) jobs below are scheduled in Program.cs. Every instance registers the same schedules, but each +// occurrence is materialized once into the shared runtime store, so scope decides how many instances run it: +// * Global (default) -> exactly ONE instance runs each tick (a leader/singleton task). +// * PerNode -> EVERY instance runs its own occurrence each tick (per-instance maintenance). +// A job uses its JobExecutionContext when it wants progress/heartbeat/identity, or ignores it (as these do). + +/// Global, every minute: a simple liveness heartbeat that runs on a single instance per tick. +public sealed class HeartbeatJob(InstanceInfo instance, ILogger logger) : IJob +{ + public Task RunAsync(JobExecutionContext context) + { + logger.LogInformation("[{Instance}] heartbeat {Time:HH:mm:ss} (one instance per tick)", instance.Id, DateTimeOffset.UtcNow); + return Task.FromResult(JobResult.Success); + } +} + +/// PerNode, every minute: each instance refreshes its own local state — so every instance runs this each tick. +public sealed class RefreshCacheJob(InstanceInfo instance, ILogger logger) : IJob +{ + public Task RunAsync(JobExecutionContext context) + { + logger.LogInformation("[{Instance}] refreshed local cache (every instance per tick)", instance.Id); + return Task.FromResult(JobResult.Success); + } +} + +/// Global, every 2 minutes: a periodic maintenance sweep that runs on a single instance per tick. +public sealed class SweepStaleOrdersJob(InstanceInfo instance, ILogger logger) : IJob +{ + public Task RunAsync(JobExecutionContext context) + { + logger.LogInformation("[{Instance}] swept stale orders (one instance per tick)", instance.Id); + return Task.FromResult(JobResult.Success); + } +} diff --git a/samples/Foundatio.MessagingSample/Messages.cs b/samples/Foundatio.MessagingSample/Messages.cs new file mode 100644 index 000000000..4196e4daf --- /dev/null +++ b/samples/Foundatio.MessagingSample/Messages.cs @@ -0,0 +1,25 @@ +using Foundatio.Messaging; + +namespace Foundatio.MessagingSample; + +/// +/// A command / unit of work, delivered with bus.SendAsync — exactly one running instance handles each one. +/// The names the destination ("orders"); without it the kebab-cased type name +/// ("process-order") is used. +/// +[MessageRoute("orders")] +public class ProcessOrder +{ + public string Product { get; set; } = ""; + public int Quantity { get; set; } = 1; +} + +/// +/// An event, delivered with bus.PublishAsync — each subscribing service receives one copy (and this sample's +/// handler opts into PerInstance, so every replica gets its own). +/// +[MessageRoute("announcements")] +public class Announcement +{ + public string Text { get; set; } = ""; +} diff --git a/samples/Foundatio.MessagingSample/Program.cs b/samples/Foundatio.MessagingSample/Program.cs new file mode 100644 index 000000000..aa6e6e6d2 --- /dev/null +++ b/samples/Foundatio.MessagingSample/Program.cs @@ -0,0 +1,57 @@ +using Foundatio; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Foundatio.MessagingSample; + +var builder = WebApplication.CreateBuilder(args); + +// A short id so log lines make it obvious WHICH instance handled each message/job when scaled to multiple replicas. +builder.Services.AddSingleton(new InstanceInfo(Guid.NewGuid().ToString("N")[..6])); + +builder.Services.AddFoundatio() + // Messaging on AWS (SQS/SNS). Handlers carry no topology decision — the caller's verb decides delivery + // (bus.SendAsync = one instance across the fleet, bus.PublishAsync = once per subscribing service). Swap UseAws() + // for UseRedis() to run messaging on Redis Streams without touching any handler. + .Messaging.UseAws() + .Messaging.AddHandler() + .Messaging.AddHandler(o => o.PerInstance = true) // every replica shows the announcement + // Durable jobs on Redis so any instance can claim them. The pump (auto-registered) runs submitted jobs and + // materializes the CRON schedules below — no manual scheduling call. + .Jobs.UseRedis() + .Jobs.Register("generate-report") // on-demand, submitted via POST /reports + .Jobs.AddCronJob("* * * * *") // Global: one instance per tick + .Jobs.AddCronJob("* * * * *", o => o.Scope = ScheduledJobScope.PerNode) // every instance per tick + .Jobs.AddCronJob("*/2 * * * *"); // Global: periodic sweep + +var app = builder.Build(); + +app.MapGet("/", (InstanceInfo instance) => Results.Ok(new { service = "Foundatio messaging sample", instance = instance.Id })); + +// SEND — a command / unit of work: exactly one instance processes each order (handled by ProcessOrderHandler). +app.MapPost("/orders", async (ProcessOrder order, IMessageBus bus) => + Results.Accepted(value: new { queued = await bus.SendAsync(order) })); + +// PUBLISH — an event: subscribers receive it per their registration (AnnouncementHandler opts into PerInstance, so +// every running replica logs each announcement). +app.MapPost("/announcements", async (Announcement announcement, IMessageBus bus) => +{ + await bus.PublishAsync(announcement); + return Results.Accepted(value: new { published = announcement.Text }); +}); + +// DURABLE JOB — submitted here, executed on whichever instance's runtime pump claims it. +app.MapPost("/reports", async (IJobClient jobs) => +{ + var handle = await jobs.EnqueueAsync(); + return Results.Accepted($"/reports/{handle.JobId}", new { jobId = handle.JobId }); +}); + +app.MapGet("/reports/{id}", async (string id, IJobMonitor monitor) => +{ + var state = await monitor.GetAsync(id); + return state is null + ? Results.NotFound() + : Results.Ok(new { state.JobId, status = state.Status.ToString(), state.Progress, state.ProgressMessage }); +}); + +app.Run(); diff --git a/samples/Foundatio.MessagingSample/Properties/launchSettings.json b/samples/Foundatio.MessagingSample/Properties/launchSettings.json new file mode 100644 index 000000000..6c31385a1 --- /dev/null +++ b/samples/Foundatio.MessagingSample/Properties/launchSettings.json @@ -0,0 +1,12 @@ +{ + "$schema": "https://json.schemastore.org/launchsettings.json", + "profiles": { + "http": { + "commandName": "Project", + "applicationUrl": "http://localhost:5280", + "environmentVariables": { + "ASPNETCORE_ENVIRONMENT": "Development" + } + } + } +} diff --git a/samples/Foundatio.MessagingSample/README.md b/samples/Foundatio.MessagingSample/README.md new file mode 100644 index 000000000..e9cd127b6 --- /dev/null +++ b/samples/Foundatio.MessagingSample/README.md @@ -0,0 +1,58 @@ +# Foundatio.MessagingSample + +A minimal ASP.NET app showing the redesigned Foundatio **messaging** (one bus, two verbs) and **durable jobs** in a +real, scaled-out setup. It runs under Aspire with **3 replicas**, so you can watch the distributed behavior. + +The core idea: **handlers are registered with no topology decision — the caller's verb decides delivery.** + +- `bus.SendAsync(msg)` — a command / unit of work: exactly **one** instance across the fleet processes it. +- `bus.PublishAsync(msg)` — an event: each subscribing **service** receives one copy (a scaled service's replicas + compete for it), or **every replica** when the handler opts in with `PerInstance = true`. + +What the sample demonstrates: + +- **Send (worker queue)** — `POST /orders` calls `bus.SendAsync`; exactly **one** replica processes each order. Scale + up and the work spreads out. +- **Publish (events)** — `POST /announcements` calls `bus.PublishAsync`; the announcement handler registers with + `PerInstance = true`, so **every** replica logs each announcement. +- **Durable job** — `POST /reports` submits a job via `IJobClient`; whichever replica's runtime pump claims it runs it. + Poll `GET /reports/{id}` to watch its status/progress. +- **CRON jobs** — declared with `.Jobs.AddCronJob(cron)` and scheduled automatically; occurrences are deduped + through the shared runtime store so **scope** decides fan-out: + - `HeartbeatJob` — Global, every minute → runs on **one** replica per tick (leader/singleton). + - `RefreshCacheJob` — PerNode, every minute → runs on **every** replica per tick (per-instance maintenance). + - `SweepStaleOrdersJob` — Global, every 2 minutes → a periodic maintenance sweep on one replica. + +Messaging runs on **AWS SQS/SNS** (via a LocalStack container) and durable jobs on **Redis** — all wired from one +clean `AddFoundatio()` chain in [`Program.cs`](Program.cs). Swap `UseAws()` for `UseRedis()` to run messaging on +Redis Streams without touching a single handler. + +## Run it (Aspire) + +```sh +dotnet run --project samples/Foundatio.AppHost +``` + +The Aspire dashboard launches Redis + LocalStack and 3 replicas of the service. Open the service endpoint and: + +```sh +# fire several orders — watch them load-balance across the 3 replicas' logs +for i in $(seq 1 6); do curl -sX POST /orders -H 'content-type: application/json' -d "{\"product\":\"widget\",\"quantity\":$i}"; done + +# publish an announcement — every replica logs it (the handler is PerInstance) +curl -sX POST /announcements -H 'content-type: application/json' -d '{"text":"hello all"}' + +# submit a durable job, then poll it +job=$(curl -sX POST /reports | jq -r .jobId); curl -s /reports/$job +``` + +The per-instance id in each log line (`[abc123] processed order: ...`) makes the distribution obvious. + +## Run it standalone (no Aspire) + +Swap `UseAws()` for `UseRedis()` in `Program.cs` (or run LocalStack for the AWS transport), point at a Redis +instance, and run: + +```sh +ConnectionStrings__Redis=localhost:6399 dotnet run --project samples/Foundatio.MessagingSample +``` diff --git a/samples/Foundatio.MessagingSample/appsettings.json b/samples/Foundatio.MessagingSample/appsettings.json new file mode 100644 index 000000000..10f68b8c8 --- /dev/null +++ b/samples/Foundatio.MessagingSample/appsettings.json @@ -0,0 +1,9 @@ +{ + "Logging": { + "LogLevel": { + "Default": "Information", + "Microsoft.AspNetCore": "Warning" + } + }, + "AllowedHosts": "*" +} diff --git a/src/Foundatio.Aws/AwsFoundatioBuilderExtensions.cs b/src/Foundatio.Aws/AwsFoundatioBuilderExtensions.cs new file mode 100644 index 000000000..d428f695d --- /dev/null +++ b/src/Foundatio.Aws/AwsFoundatioBuilderExtensions.cs @@ -0,0 +1,46 @@ +using System; +using Amazon; +using Amazon.Runtime; +using Foundatio.Messaging; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; + +namespace Foundatio; + +public static class AwsFoundatioBuilderExtensions +{ + /// + /// Runs messaging (queues on SQS, pub/sub on SNS+SQS) over AWS. With no arguments it relies on the AWS SDK's default + /// region and credential resolution; common settings (ServiceUrl, Region, ResourcePrefix, AccessKey/SecretKey) are + /// also bound from an "Aws" configuration section when present, and can override + /// anything. Point ServiceUrl at LocalStack to run without a cloud account. + /// + public static FoundatioBuilder UseAws(this FoundatioBuilder.MessagingBuilder builder, Action? configure = null) + { + return builder.UseTransport(sp => + { + var options = new AwsMessageTransportOptions(); + BindFromConfiguration(options, sp.GetService()?.GetSection("Aws")); + configure?.Invoke(options); + return new AwsMessageTransport(options); + }); + } + + private static void BindFromConfiguration(AwsMessageTransportOptions options, IConfiguration? section) + { + if (section is null) + return; + + if (section["ServiceUrl"] is { Length: > 0 } serviceUrl) + options.ServiceUrl = serviceUrl; + + if (section["Region"] is { Length: > 0 } region) + options.Region = RegionEndpoint.GetBySystemName(region); + + if (section["ResourcePrefix"] is { Length: > 0 } prefix) + options.ResourcePrefix = prefix; + + if (section["AccessKey"] is { Length: > 0 } accessKey && section["SecretKey"] is { Length: > 0 } secretKey) + options.Credentials = new BasicAWSCredentials(accessKey, secretKey); + } +} diff --git a/src/Foundatio.Aws/AwsMessageTransport.cs b/src/Foundatio.Aws/AwsMessageTransport.cs new file mode 100644 index 000000000..039fe599a --- /dev/null +++ b/src/Foundatio.Aws/AwsMessageTransport.cs @@ -0,0 +1,577 @@ +using System; +using System.Collections.Generic; +using System.Collections.Concurrent; +using System.Linq; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Amazon.SQS; +using Amazon.SQS.Model; +using Amazon.SimpleNotificationService; +using Amazon.SimpleNotificationService.Model; +using SnsMessageAttributeValue = Amazon.SimpleNotificationService.Model.MessageAttributeValue; +using SqsMessageAttributeValue = Amazon.SQS.Model.MessageAttributeValue; +using SqsMessage = Amazon.SQS.Model.Message; + +namespace Foundatio.Messaging; + +/// +/// An over AWS SQS (queues + competing-consumer subscriptions) and SNS (topics). This +/// is a temporary in-repo provider used to validate the redesigned transport contract against a real broker. Queue and +/// subscription destinations are SQS queues; topic destinations are SNS topics fanned out to SQS subscription queues. +/// +/// +/// Capability mapping: pull receive (SQS long poll), visibility timeout, redelivery delay (ChangeMessageVisibility, +/// 12h cap), delayed delivery on queues only (SQS DelaySeconds, 15-minute cap — SNS topics have no native delay, so +/// delayed publishes route through the runtime-store fallback), provisioning, and stats. SQS has no per-message +/// priority, per-message TTL, or push delivery, and no transport-native dead-letter that the core controls the timing +/// of, so those capabilities are intentionally not implemented (the core owns retry/dead-lettering). +/// +public sealed class AwsMessageTransport : IMessageTransport, ISupportsPull, ISupportsVisibilityTimeout, + ISupportsLockRenewal, ISupportsRedeliveryDelay, ISupportsProvisioning, ISupportsStats, ITransportInfo +{ + private const string HeadersAttributeName = "fnd.headers"; + private const string EncodingAttributeName = "fnd.encoding"; + + // Well-known headers surfaced as native message attributes (in addition to the authoritative JSON blob) so brokers + // can filter/route on them — e.g. SNS subscription filter policies match on native attributes. + private static readonly string[] WellKnownNativeHeaders = [KnownHeaders.MessageType, KnownHeaders.Priority, KnownHeaders.CorrelationId]; + + private static readonly IReadOnlySet _supportedRoles = + new HashSet { DestinationRole.Queue, DestinationRole.Topic, DestinationRole.Subscription, DestinationRole.Binding }; + + private readonly AwsMessageTransportOptions _options; + private readonly Lazy _sqs; + private readonly Lazy _sns; + private readonly ConcurrentDictionary _queueUrls = new(StringComparer.Ordinal); + private readonly ConcurrentDictionary _topicArns = new(StringComparer.Ordinal); + private int _isDisposed; + + public AwsMessageTransport(AwsMessageTransportOptions options) + { + _options = options ?? throw new ArgumentNullException(nameof(options)); + _sqs = new Lazy(CreateSqsClient); + _sns = new Lazy(CreateSnsClient); + } + + public AwsMessageTransport(string connectionString) : this(AwsMessageTransportOptions.FromConnectionString(connectionString)) { } + + // Capabilities differ by role: SQS queues take a native DelaySeconds (15-minute cap), SNS topics have no native + // delay at all — a delayed publish must route through the runtime-store fallback, never silently drop the delay. + // The 256 KB body limit applies to both services. + private static readonly TransportCapabilities _queueCapabilities = new() + { + DelayedDelivery = true, + MaxDeliveryDelay = TimeSpan.FromMinutes(15), // SQS DelaySeconds maximum + MaxMessageBytes = 262144 // 256 KB SQS limit + }; + + private static readonly TransportCapabilities _topicCapabilities = new() + { + MaxMessageBytes = 262144 // 256 KB SNS limit + }; + + public DeliveryGuarantee DeliveryGuarantee => DeliveryGuarantee.AtLeastOnce; + public IReadOnlySet SupportedRoles => _supportedRoles; + + public TransportCapabilities GetCapabilities(DestinationRole role) => + role == DestinationRole.Topic ? _topicCapabilities : _queueCapabilities; + + public TimeSpan? MaxRedeliveryDelay => TimeSpan.FromHours(12); // SQS ChangeMessageVisibility maximum + public TimeSpan? MaxVisibilityTimeout => TimeSpan.FromHours(12); // SQS visibility maximum + + public async Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(destination); + ArgumentNullException.ThrowIfNull(messages); + + var items = new List(messages.Count); + + // The address states the destination role, so route without inferring: a topic publishes to SNS, anything else + // sends to an SQS queue. + if (destination.Role == DestinationRole.Topic) + { + // SNS has no native delayed publish. The core routes delayed topic publishes through the runtime-store + // fallback (topic capabilities advertise no DelayedDelivery), so a DeliverAt reaching here is a contract + // violation — refuse loudly rather than publish immediately and silently drop the delay. + if (options.DeliverAt is { } deliverAt && deliverAt > DateTimeOffset.UtcNow) + throw new NotSupportedException($"Transport \"{nameof(AwsMessageTransport)}\" does not support delayed delivery for Topic destinations (SNS has no native delay). Register a job runtime store so delayed publishes use the scheduled-dispatch fallback."); + + string topicArn = await ResolveTopicArnAsync(destination.Name, ct).ConfigureAwait(false); + foreach (var message in messages) + { + var (body, encoding) = EncodeBody(message); + var response = await _sns.Value.PublishAsync(new PublishRequest + { + TopicArn = topicArn, + Message = body, + MessageAttributes = BuildAttributes(message.Headers, encoding, static value => new SnsMessageAttributeValue { DataType = "String", StringValue = value }) + }, ct).ConfigureAwait(false); + + items.Add(new SendItemResult { MessageId = response.MessageId }); + } + + return new SendResult { Items = items }; + } + + string queueUrl = await ResolveQueueUrlAsync(destination, ct).ConfigureAwait(false); + int? delaySeconds = ToDelaySeconds(options.DeliverAt); + foreach (var message in messages) + { + var (body, encoding) = EncodeBody(message); + var request = new SendMessageRequest + { + QueueUrl = queueUrl, + MessageBody = body, + MessageAttributes = BuildAttributes(message.Headers, encoding, static value => new SqsMessageAttributeValue { DataType = "String", StringValue = value }) + }; + if (delaySeconds is { } delay) + request.DelaySeconds = delay; + + var response = await _sqs.Value.SendMessageAsync(request, ct).ConfigureAwait(false); + items.Add(new SendItemResult { MessageId = response.MessageId }); + } + + return new SendResult { Items = items }; + } + + public Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct) + { + return ReceiveAsync(source, request, _options.DefaultVisibilityTimeout, ct); + } + + public async Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, TimeSpan visibility, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(source); + ArgumentNullException.ThrowIfNull(request); + + string queueUrl = await ResolveQueueUrlAsync(source, ct).ConfigureAwait(false); + + var sqsRequest = new ReceiveMessageRequest + { + QueueUrl = queueUrl, + MaxNumberOfMessages = Math.Clamp(request.MaxMessages <= 0 ? 1 : request.MaxMessages, 1, 10), + VisibilityTimeout = (int)Math.Clamp(visibility.TotalSeconds, 0, 43200), + MessageAttributeNames = ["All"], + MessageSystemAttributeNames = ["All"] + }; + if (request.MaxWaitTime is { } wait) + sqsRequest.WaitTimeSeconds = (int)Math.Clamp(wait.TotalSeconds, 0, 20); + + var response = await _sqs.Value.ReceiveMessageAsync(sqsRequest, ct).ConfigureAwait(false); + if (response.Messages is not { Count: > 0 }) + return []; + + var entries = new List(response.Messages.Count); + foreach (var message in response.Messages) + { + entries.Add(new TransportEntry + { + Id = message.MessageId, + Destination = source, + Body = DecodeBody(message.Body, GetAttribute(message.MessageAttributes, EncodingAttributeName)), + Headers = FromSqsAttributes(message.MessageAttributes), + DeliveryCount = GetReceiveCount(message), + Receipt = new Receipt { TransportState = message.ReceiptHandle } + }); + } + + return entries; + } + + public async Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) + { + ThrowIfDisposed(); + string queueUrl = await ResolveQueueUrlAsync(entry.Destination, ct).ConfigureAwait(false); + await _sqs.Value.DeleteMessageAsync(queueUrl, GetReceiptHandle(entry), ct).ConfigureAwait(false); + } + + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) + { + return AbandonAsync(entry, TimeSpan.Zero, ct); + } + + public async Task AbandonAsync(TransportEntry entry, TimeSpan redeliveryDelay, CancellationToken ct) + { + ThrowIfDisposed(); + string queueUrl = await ResolveQueueUrlAsync(entry.Destination, ct).ConfigureAwait(false); + // Returning a message to the queue is a visibility change to the requested delay (0 = immediately visible). + await _sqs.Value.ChangeMessageVisibilityAsync(queueUrl, GetReceiptHandle(entry), (int)Math.Clamp(redeliveryDelay.TotalSeconds, 0, 43200), ct).ConfigureAwait(false); + } + + public async Task RenewLockAsync(TransportEntry entry, TimeSpan? duration, CancellationToken ct) + { + ThrowIfDisposed(); + string queueUrl = await ResolveQueueUrlAsync(entry.Destination, ct).ConfigureAwait(false); + int seconds = (int)Math.Clamp((duration ?? _options.DefaultVisibilityTimeout).TotalSeconds, 0, 43200); + await _sqs.Value.ChangeMessageVisibilityAsync(queueUrl, GetReceiptHandle(entry), seconds, ct).ConfigureAwait(false); + } + + public async Task EnsureAsync(IReadOnlyList declarations, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(declarations); + + foreach (var declaration in declarations) + { + switch (declaration.Address.Role) + { + case DestinationRole.Topic: + await ResolveTopicArnAsync(declaration.Address.Name, allowCreate: true, ct).ConfigureAwait(false); + break; + case DestinationRole.Subscription: + case DestinationRole.Binding: + await EnsureSubscriptionAsync(declaration.Address, ct).ConfigureAwait(false); + break; + default: + await ResolveQueueUrlAsync(declaration.Address, allowCreate: true, ct).ConfigureAwait(false); + break; + } + } + } + + public async Task DeleteAsync(DestinationAddress destination, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(destination); + + if (destination.Role == DestinationRole.Topic) + { + if (_topicArns.TryRemove(destination.Name, out string? arn)) + await _sns.Value.DeleteTopicAsync(arn, ct).ConfigureAwait(false); + return; + } + + // Queue and subscription destinations are both backed by an SQS queue named from the address key. + if (_queueUrls.TryRemove(destination.Key, out string? url)) + await _sqs.Value.DeleteQueueAsync(url, ct).ConfigureAwait(false); + } + + public async Task ExistsAsync(DestinationAddress destination, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(destination); + + if (destination.Role == DestinationRole.Topic) + return _topicArns.ContainsKey(destination.Name); + + try + { + await _sqs.Value.GetQueueUrlAsync(ResourceName(destination.Key), ct).ConfigureAwait(false); + return true; + } + catch (QueueDoesNotExistException) + { + return false; + } + } + + public async Task GetStatsAsync(DestinationAddress destination, CancellationToken ct) + { + ThrowIfDisposed(); + string queueUrl = await ResolveQueueUrlAsync(destination, ct).ConfigureAwait(false); + var response = await _sqs.Value.GetQueueAttributesAsync(new GetQueueAttributesRequest + { + QueueUrl = queueUrl, + AttributeNames = ["All"] + }, ct).ConfigureAwait(false); + + return new MessageDestinationStats + { + Queued = response.ApproximateNumberOfMessages, + Working = response.ApproximateNumberOfMessagesNotVisible + }; + } + + public async ValueTask DisposeAsync() + { + if (Interlocked.Exchange(ref _isDisposed, 1) == 1) + return; + + if (_sqs.IsValueCreated) + _sqs.Value.Dispose(); + if (_sns.IsValueCreated) + _sns.Value.Dispose(); + + await ValueTask.CompletedTask.ConfigureAwait(false); + } + + private async Task EnsureSubscriptionAsync(DestinationAddress address, CancellationToken ct) + { + string queueUrl = await ResolveQueueUrlAsync(address, allowCreate: true, ct).ConfigureAwait(false); + + if (String.IsNullOrEmpty(address.Topic)) + return; + + string topicArn = await ResolveTopicArnAsync(address.Topic, allowCreate: true, ct).ConfigureAwait(false); + string queueArn = await GetQueueArnAsync(queueUrl, ct).ConfigureAwait(false); + + // Allow the topic to deliver to the queue, then subscribe with raw delivery so the SQS body/attributes match a + // direct SQS send (no SNS envelope). + await _sqs.Value.SetQueueAttributesAsync(new SetQueueAttributesRequest + { + QueueUrl = queueUrl, + Attributes = new Dictionary { ["Policy"] = BuildQueuePolicy(queueArn, topicArn) } + }, ct).ConfigureAwait(false); + + await _sns.Value.SubscribeAsync(new SubscribeRequest + { + TopicArn = topicArn, + Protocol = "sqs", + Endpoint = queueArn, + Attributes = new Dictionary { ["RawMessageDelivery"] = "true" }, + ReturnSubscriptionArn = true + }, ct).ConfigureAwait(false); + } + + // Queue and subscription destinations are both backed by an SQS queue whose logical name is the address key + // (Name for queues, "topic/subscription" for subscriptions), so provisioning and every runtime path resolve the + // same physical queue from the same address. + private Task ResolveQueueUrlAsync(DestinationAddress address, CancellationToken ct) => + ResolveQueueUrlAsync(address, allowCreate: _options.AutoCreateDestinations, ct); + + private async Task ResolveQueueUrlAsync(DestinationAddress address, bool allowCreate, CancellationToken ct) + { + string key = address.Key; + if (_queueUrls.TryGetValue(key, out string? cached)) + return cached; + + string resourceName = ResourceName(key); + try + { + var response = await _sqs.Value.GetQueueUrlAsync(resourceName, ct).ConfigureAwait(false); + _queueUrls[key] = response.QueueUrl; + return response.QueueUrl; + } + catch (QueueDoesNotExistException) when (allowCreate) + { + var response = await _sqs.Value.CreateQueueAsync(new CreateQueueRequest { QueueName = resourceName }, ct).ConfigureAwait(false); + _queueUrls[key] = response.QueueUrl; + return response.QueueUrl; + } + } + + // Implicit resolution (send/receive paths) honors AutoCreateDestinations; explicit provisioning via EnsureAsync + // always creates — that call IS the administrative intent the option exists to withhold from the data paths. + private Task ResolveTopicArnAsync(string name, CancellationToken ct) => + ResolveTopicArnAsync(name, allowCreate: _options.AutoCreateDestinations, ct); + + private async Task ResolveTopicArnAsync(string name, bool allowCreate, CancellationToken ct) + { + if (_topicArns.TryGetValue(name, out string? cached)) + return cached; + + if (allowCreate) + { + // CreateTopic is idempotent and returns the ARN of an existing topic with the same name. + var response = await _sns.Value.CreateTopicAsync(new CreateTopicRequest { Name = ResourceName(name) }, ct).ConfigureAwait(false); + _topicArns[name] = response.TopicArn; + return response.TopicArn; + } + + // Auto-create is disabled (locked-down broker): look the topic up instead of creating it, and fail loudly when + // it has not been provisioned out of band. + var existing = await _sns.Value.FindTopicAsync(ResourceName(name)).ConfigureAwait(false); + if (existing is null) + throw new InvalidOperationException($"SNS topic \"{ResourceName(name)}\" does not exist and {nameof(AwsMessageTransportOptions.AutoCreateDestinations)} is disabled. Provision it out of band or enable auto-creation."); + + _topicArns[name] = existing.TopicArn; + return existing.TopicArn; + } + + // SQS queue / SNS topic names allow only [A-Za-z0-9_-] (max 80 chars). Most logical names already conform, but a + // subscription's key (see DestinationAddress.Key) is the opaque "topic/subscription" form which contains '/'. + // Encode any illegal name deterministically and collision-free — sanitize, then append a short stable hash of the + // original — so EnsureAsync/ReceiveAsync/CompleteAsync all resolve the same queue from the same logical name. + // Legal names are returned unchanged (no behavior change for plain queues/topics). + private string ResourceName(string logicalName) => EncodeResourceName(_options.ResourcePrefix, logicalName); + + private static string EncodeResourceName(string prefix, string logicalName) + { + string candidate = prefix + logicalName; + if (IsResourceNameLegal(candidate)) + return candidate; + + string suffix = "-" + StableHash(candidate); + string sanitized = SanitizeResourceName(candidate); + if (sanitized.Length > 80 - suffix.Length) + sanitized = sanitized[..(80 - suffix.Length)]; + return sanitized + suffix; + } + + private static bool IsResourceNameLegal(string name) + { + if (name.Length is 0 or > 80) + return false; + foreach (char c in name) + { + if (!(Char.IsAsciiLetterOrDigit(c) || c is '-' or '_')) + return false; + } + + return true; + } + + private static string SanitizeResourceName(string name) + { + var builder = new StringBuilder(name.Length); + foreach (char c in name) + builder.Append(Char.IsAsciiLetterOrDigit(c) || c is '-' or '_' ? c : '-'); + return builder.ToString(); + } + + private static string StableHash(string value) + { + byte[] hash = SHA256.HashData(Encoding.UTF8.GetBytes(value)); + return Convert.ToHexString(hash, 0, 4).ToLowerInvariant(); // 8 hex chars + } + + private async Task GetQueueArnAsync(string queueUrl, CancellationToken ct) + { + var response = await _sqs.Value.GetQueueAttributesAsync(new GetQueueAttributesRequest + { + QueueUrl = queueUrl, + AttributeNames = ["QueueArn"] + }, ct).ConfigureAwait(false); + return response.QueueARN; + } + + private static string BuildQueuePolicy(string queueArn, string topicArn) + { + return JsonSerializer.Serialize(new + { + Version = "2012-10-17", + Statement = new[] + { + new + { + Effect = "Allow", + Principal = new { Service = "sns.amazonaws.com" }, + Action = "sqs:SendMessage", + Resource = queueArn, + Condition = new { ArnEquals = new Dictionary { ["aws:SourceArn"] = topicArn } } + } + } + }); + } + + private int? ToDelaySeconds(DateTimeOffset? deliverAt) + { + if (deliverAt is not { } at) + return null; + + double seconds = (at - DateTimeOffset.UtcNow).TotalSeconds; + if (seconds <= 0) + return null; + + return (int)Math.Clamp(seconds, 1, 900); // SQS DelaySeconds maximum is 900 (15 minutes) + } + + private static int GetReceiveCount(SqsMessage message) + { + if (message.Attributes is not null && message.Attributes.TryGetValue("ApproximateReceiveCount", out string? value) && Int32.TryParse(value, out int count) && count > 0) + return count; + return 1; + } + + private static string GetReceiptHandle(TransportEntry entry) + { + return entry.Receipt.TransportState as string + ?? throw new ReceiptExpiredException("The transport entry does not carry an SQS receipt handle."); + } + + // A text body (e.g. JSON, the default) is stored as-is so it is human-readable in the console and avoids base64 + // overhead; anything else is base64-encoded so arbitrary bytes round-trip through SQS/SNS string bodies. The chosen + // encoding is recorded in a native attribute for the receive side. + private static (string Body, string Encoding) EncodeBody(TransportMessage message) + { + return IsTextContent(message.ContentType) + ? (Encoding.UTF8.GetString(message.Body.Span), "text") + : (Convert.ToBase64String(message.Body.Span), "base64"); + } + + private static ReadOnlyMemory DecodeBody(string body, string? encoding) + { + if (String.IsNullOrEmpty(body)) + return ReadOnlyMemory.Empty; + + return String.Equals(encoding, "text", StringComparison.Ordinal) + ? Encoding.UTF8.GetBytes(body) + : Convert.FromBase64String(body); + } + + private static bool IsTextContent(string? contentType) + { + return !String.IsNullOrEmpty(contentType) + && (contentType.Contains("json", StringComparison.OrdinalIgnoreCase) + || contentType.Contains("xml", StringComparison.OrdinalIgnoreCase) + || contentType.StartsWith("text/", StringComparison.OrdinalIgnoreCase)); + } + + private static Dictionary BuildAttributes(MessageHeaders headers, string encoding, Func stringAttribute) + { + var attributes = new Dictionary(StringComparer.Ordinal) + { + [HeadersAttributeName] = stringAttribute(MessageHeaders.SerializeToJson(headers)), + [EncodingAttributeName] = stringAttribute(encoding) + }; + + foreach (string name in WellKnownNativeHeaders) + { + string? value = headers.GetValueOrDefault(name); + if (!String.IsNullOrEmpty(value)) + attributes[name] = stringAttribute(value); + } + + return attributes; + } + + private static string? GetAttribute(Dictionary? attributes, string name) + { + return attributes is not null && attributes.TryGetValue(name, out var value) ? value.StringValue : null; + } + + private static MessageHeaders FromSqsAttributes(Dictionary? attributes) + { + if (attributes is null || !attributes.TryGetValue(HeadersAttributeName, out var value) || String.IsNullOrEmpty(value.StringValue)) + return MessageHeaders.Empty; + + return MessageHeaders.DeserializeFromJson(value.StringValue); + } + + private IAmazonSQS CreateSqsClient() + { + var config = new AmazonSQSConfig(); + ApplyEndpoint(config); + return _options.Credentials is { } credentials ? new AmazonSQSClient(credentials, config) : new AmazonSQSClient(config); + } + + private IAmazonSimpleNotificationService CreateSnsClient() + { + var config = new AmazonSimpleNotificationServiceConfig(); + ApplyEndpoint(config); + return _options.Credentials is { } credentials ? new AmazonSimpleNotificationServiceClient(credentials, config) : new AmazonSimpleNotificationServiceClient(config); + } + + private void ApplyEndpoint(Amazon.Runtime.ClientConfig config) + { + if (!String.IsNullOrEmpty(_options.ServiceUrl)) + { + config.ServiceURL = _options.ServiceUrl; + config.AuthenticationRegion = (_options.Region ?? Amazon.RegionEndpoint.USEast1).SystemName; + } + else if (_options.Region is { } region) + { + config.RegionEndpoint = region; + } + } + + private void ThrowIfDisposed() + { + ObjectDisposedException.ThrowIf(Volatile.Read(ref _isDisposed) == 1, this); + } +} diff --git a/src/Foundatio.Aws/AwsMessageTransportOptions.cs b/src/Foundatio.Aws/AwsMessageTransportOptions.cs new file mode 100644 index 000000000..8b02fd267 --- /dev/null +++ b/src/Foundatio.Aws/AwsMessageTransportOptions.cs @@ -0,0 +1,80 @@ +using System; +using Amazon; +using Amazon.Runtime; + +namespace Foundatio.Messaging; + +public class AwsMessageTransportOptions +{ + /// AWS credentials. When null, the SDK's default credential chain is used. + public AWSCredentials? Credentials { get; set; } + + /// AWS region. When null, the SDK's default region resolution is used (ignored when is set). + public RegionEndpoint? Region { get; set; } + + /// Custom service endpoint, e.g. http://localhost:4566 for LocalStack. + public string? ServiceUrl { get; set; } + + /// Create queues/topics/subscriptions on demand when sending or receiving (in addition to explicit provisioning). + public bool AutoCreateDestinations { get; set; } = true; + + /// + /// Optional prefix applied to the underlying SQS queue and SNS topic names (not the logical destination names used + /// by callers). Useful to isolate runs/environments on a shared broker — e.g. a unique prefix per conformance run + /// so leftover messages from a prior run can't leak in. + /// + public string ResourcePrefix { get; set; } = ""; + + /// Default receive visibility timeout when none is supplied. Maps to the SQS visibility window. + public TimeSpan DefaultVisibilityTimeout { get; set; } = TimeSpan.FromSeconds(30); + + /// + /// Parses a connection string of the form + /// serviceurl=http://localhost:4566;accesskey=...;secretkey=...;region=us-east-1 into options. Any subset of + /// keys may be provided; unknown keys are ignored. + /// + public static AwsMessageTransportOptions FromConnectionString(string connectionString) + { + ArgumentException.ThrowIfNullOrEmpty(connectionString); + + string? accessKey = null, secretKey = null, region = null, serviceUrl = null; + foreach (string pair in connectionString.Split(';', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries)) + { + int separator = pair.IndexOf('='); + if (separator < 0) + continue; + + string key = pair[..separator].Trim().ToLowerInvariant().Replace(" ", ""); + string value = pair[(separator + 1)..].Trim(); + + switch (key) + { + case "accesskey": + case "accesskeyid": + case "id": + accessKey = value; + break; + case "secretkey": + case "secret": + secretKey = value; + break; + case "region": + case "endpoint": + region = value; + break; + case "serviceurl": + case "service": + serviceUrl = value; + break; + } + } + + var options = new AwsMessageTransportOptions { ServiceUrl = serviceUrl }; + if (!String.IsNullOrEmpty(accessKey) && !String.IsNullOrEmpty(secretKey)) + options.Credentials = new BasicAWSCredentials(accessKey, secretKey); + if (!String.IsNullOrEmpty(region)) + options.Region = RegionEndpoint.GetBySystemName(region); + + return options; + } +} diff --git a/src/Foundatio.Aws/Foundatio.Aws.csproj b/src/Foundatio.Aws/Foundatio.Aws.csproj new file mode 100644 index 000000000..2c8635702 --- /dev/null +++ b/src/Foundatio.Aws/Foundatio.Aws.csproj @@ -0,0 +1,12 @@ + + + AWS (SQS/SNS) IMessageTransport for Foundatio messaging. Temporary in-repo provider for validating the redesigned transport contract against a real broker. + + + + + + + + + diff --git a/src/Foundatio.Extensions.Hosting/Jobs/Cron.cs b/src/Foundatio.Extensions.Hosting/Jobs/Cron.cs index ffc193481..e13bbcb5a 100644 --- a/src/Foundatio.Extensions.Hosting/Jobs/Cron.cs +++ b/src/Foundatio.Extensions.Hosting/Jobs/Cron.cs @@ -15,7 +15,7 @@ using System; -namespace Foundatio.Extensions.Hosting.Jobs; +namespace Foundatio.Extensions.Hosting.Jobs.Legacy; /// /// Helper class that provides common values for the cron expressions. diff --git a/src/Foundatio.Extensions.Hosting/Jobs/DynamicJob.cs b/src/Foundatio.Extensions.Hosting/Jobs/DynamicJob.cs index 1eda91928..cfe7ee563 100644 --- a/src/Foundatio.Extensions.Hosting/Jobs/DynamicJob.cs +++ b/src/Foundatio.Extensions.Hosting/Jobs/DynamicJob.cs @@ -1,10 +1,10 @@ using System; using System.Threading; using System.Threading.Tasks; -using Foundatio.Jobs; +using Foundatio.Jobs.Legacy; using Foundatio.Utility; -namespace Foundatio.Extensions.Hosting.Jobs; +namespace Foundatio.Extensions.Hosting.Jobs.Legacy; internal class DynamicJob : IJob { diff --git a/src/Foundatio.Extensions.Hosting/Jobs/HostedJobOptions.cs b/src/Foundatio.Extensions.Hosting/Jobs/HostedJobOptions.cs index ce9488ed7..664065cf1 100644 --- a/src/Foundatio.Extensions.Hosting/Jobs/HostedJobOptions.cs +++ b/src/Foundatio.Extensions.Hosting/Jobs/HostedJobOptions.cs @@ -1,8 +1,8 @@ using Foundatio.Jobs; -namespace Foundatio.Extensions.Hosting.Jobs; +namespace Foundatio.Extensions.Hosting.Jobs.Legacy; -public class HostedJobOptions : JobOptions +public class HostedJobOptions : Foundatio.Jobs.Legacy.JobOptions { public bool WaitForStartupActions { get; set; } } diff --git a/src/Foundatio.Extensions.Hosting/Jobs/HostedJobService.cs b/src/Foundatio.Extensions.Hosting/Jobs/HostedJobService.cs index d5d7b2eba..48867dd00 100644 --- a/src/Foundatio.Extensions.Hosting/Jobs/HostedJobService.cs +++ b/src/Foundatio.Extensions.Hosting/Jobs/HostedJobService.cs @@ -2,13 +2,13 @@ using System.Threading; using System.Threading.Tasks; using Foundatio.Extensions.Hosting.Startup; -using Foundatio.Jobs; +using Foundatio.Jobs.Legacy; using Foundatio.Utility; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Logging; -namespace Foundatio.Extensions.Hosting.Jobs; +namespace Foundatio.Extensions.Hosting.Jobs.Legacy; public class HostedJobService : IHostedService, IJobStatus, IDisposable { diff --git a/src/Foundatio.Extensions.Hosting/Jobs/JobHostExtensions.cs b/src/Foundatio.Extensions.Hosting/Jobs/JobHostExtensions.cs index 1a00ac6c0..9d53b68bf 100644 --- a/src/Foundatio.Extensions.Hosting/Jobs/JobHostExtensions.cs +++ b/src/Foundatio.Extensions.Hosting/Jobs/JobHostExtensions.cs @@ -1,203 +1,34 @@ -using System; -using System.Linq; -using System.Threading; -using System.Threading.Tasks; +using System; using Foundatio.Jobs; using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Hosting; -using Microsoft.Extensions.Logging; namespace Foundatio.Extensions.Hosting.Jobs; public static class JobHostExtensions { - public static IServiceCollection AddJob(this IServiceCollection services, HostedJobOptions jobOptions) - { - if (jobOptions.JobFactory == null) - throw new ArgumentNullException(nameof(jobOptions), "jobOptions.JobFactory is required"); - - return services.AddTransient(s => new HostedJobService(s, jobOptions, s.GetRequiredService())); - } - - public static IServiceCollection AddJob(this IServiceCollection services, HostedJobOptions? jobOptions = null) where T : class, IJob - { - services.AddTransient(); - return services.AddTransient(s => + /// + /// Registers the hosted pump that drives the durable job runtime (): + /// materializing CRON occurrences, dispatching delayed/scheduled work, recovering stale occurrences, and running + /// jobs submitted via . Register the runtime store and job services first + /// (e.g. services.AddFoundatio().Jobs.UseInMemoryRuntime()). + /// + public static IServiceCollection AddJobRuntimeService(this IServiceCollection services, Action? configure = null) + { + var options = new JobRuntimeServiceOptions(); + configure?.Invoke(options); + + // Registering a runtime store (AddFoundatio().Jobs.UseRuntimeStore()/UseInMemoryRuntime()) is the precondition + // for this call, and that already auto-registers the single runtime pump (JobRuntimePumpService). So this method + // only carries options onto that pump — it never starts a second pump — which keeps a single pump regardless of + // the order AddJobRuntimeService and UseRuntimeStore are called in. + services.AddSingleton(new JobRuntimePumpOptions { - if (jobOptions == null) - { - jobOptions = new HostedJobOptions(); - jobOptions.ApplyDefaults(); - } - - jobOptions.Name ??= JobOptions.GetDefaultJobName(typeof(T)); - jobOptions.JobFactory ??= sp => sp.GetRequiredService(); - - return new HostedJobService(s, jobOptions, s.GetRequiredService()); + Enabled = options.Enabled, + PollInterval = options.PollInterval, + BatchSize = options.BatchSize, + MaxJobAttempts = options.MaxJobAttempts }); - } - - public static IServiceCollection AddJob(this IServiceCollection services, Action configureJobOptions) where T : class, IJob - { - var jobOptionsBuilder = new HostedJobOptionsBuilder(); - jobOptionsBuilder.ApplyDefaults(); - jobOptionsBuilder.Name(JobOptions.GetDefaultJobName(typeof(T))); - configureJobOptions?.Invoke(jobOptionsBuilder); - return services.AddJob(jobOptionsBuilder.Target); - } - - public static IServiceCollection AddJob(this IServiceCollection services, Action configureJobOptions) - { - var jobOptionsBuilder = new HostedJobOptionsBuilder(); - configureJobOptions?.Invoke(jobOptionsBuilder); - return services.AddJob(jobOptionsBuilder.Target); - } - - public static IServiceCollection AddJob(this IServiceCollection services, string name, Func jobFactory, Action configureJobOptions) - { - var jobOptionsBuilder = new HostedJobOptionsBuilder(); - jobOptionsBuilder.Name(name).JobFactory(jobFactory); - configureJobOptions?.Invoke(jobOptionsBuilder); - return services.AddJob(jobOptionsBuilder.Target); - } - - public static IServiceCollection AddCronJob(this IServiceCollection services, ScheduledJobOptions jobOptions) - { - if (jobOptions.JobFactory == null) - throw new ArgumentNullException(nameof(jobOptions), "jobOptions.JobFactory is required"); - - services.AddJobScheduler(); - - return services.AddTransient(s => new ScheduledJobRegistration(jobOptions)); - } - - public static IServiceCollection AddCronJob(this IServiceCollection services, Action configureJobOptions) - { - var jobOptionsBuilder = new ScheduledJobOptionsBuilder(); - configureJobOptions?.Invoke(jobOptionsBuilder); - return services.AddCronJob(jobOptionsBuilder.Target); - } - - public static IServiceCollection AddCronJob(this IServiceCollection services, string cronSchedule, Action? configureJobOptions = null) where T : class, IJob - { - services.AddTransient(); - var jobOptionsBuilder = new ScheduledJobOptionsBuilder(); - jobOptionsBuilder.Name(JobOptions.GetDefaultJobName(typeof(T))).CronSchedule(cronSchedule).JobFactory(sp => sp.GetRequiredService()); - configureJobOptions?.Invoke(jobOptionsBuilder); - return services.AddCronJob(jobOptionsBuilder.Target); - } - - public static IServiceCollection AddCronJob(this IServiceCollection services, string name, string cronSchedule, Func action) - { - return services.AddCronJob(o => o.Name(name).CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, action))); - } - - public static IServiceCollection AddCronJob(this IServiceCollection services, string name, string cronSchedule, Func action) - { - return services.AddCronJob(o => o.Name(name).CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (xp, _) => action(xp)))); - } - - public static IServiceCollection AddCronJob(this IServiceCollection services, string name, string cronSchedule, Func action) - { - return services.AddCronJob(o => o.Name(name).CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (_, _) => action()))); - } - - public static IServiceCollection AddCronJob(this IServiceCollection services, string name, string cronSchedule, Action action) - { - return services.AddCronJob(o => o.Name(name).CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (xp, ct) => - { - action(xp, ct); - return Task.CompletedTask; - }))); - } - public static IServiceCollection AddCronJob(this IServiceCollection services, string name, string cronSchedule, Action action) - { - return services.AddCronJob(o => o.Name(name).CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (_, ct) => - { - action(ct); - return Task.CompletedTask; - }))); - } - - public static IServiceCollection AddCronJob(this IServiceCollection services, string name, string cronSchedule, Action action) - { - return services.AddCronJob(o => o.Name(name).CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (_, _) => - { - action(); - return Task.CompletedTask; - }))); - } - - public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string cronSchedule, Action? configureJobOptions = null) where T : class, IJob - { - services.AddTransient(); - var jobOptionsBuilder = new ScheduledJobOptionsBuilder(); - jobOptionsBuilder.Name(JobOptions.GetDefaultJobName(typeof(T))).Distributed().CronSchedule(cronSchedule).JobFactory(sp => sp.GetRequiredService()); - configureJobOptions?.Invoke(jobOptionsBuilder); - return services.AddCronJob(jobOptionsBuilder.Target); - } - - public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string name, string cronSchedule, Func action) - { - return services.AddCronJob(o => o.Name(name).Distributed().CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, action))); - } - - public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string name, string cronSchedule, Func action) - { - return services.AddCronJob(o => o.Name(name).Distributed().CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (xp, _) => action(xp)))); - } - - public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string name, string cronSchedule, Func action) - { - return services.AddCronJob(o => o.Name(name).Distributed().CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (_, _) => action()))); - } - - public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string name, string cronSchedule, Action action) - { - return services.AddCronJob(o => o.Name(name).Distributed().CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (xp, ct) => - { - action(xp, ct); - return Task.CompletedTask; - }))); - } - - public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string name, string cronSchedule, Action action) - { - return services.AddCronJob(o => o.Name(name).Distributed().CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (_, ct) => - { - action(ct); - return Task.CompletedTask; - }))); - } - - public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string name, string cronSchedule, Action action) - { - return services.AddCronJob(o => o.Name(name).Distributed().CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (_, _) => - { - action(); - return Task.CompletedTask; - }))); - } - - public static IServiceCollection AddJobScheduler(this IServiceCollection services) - { - if (!services.Any(s => s.ServiceType == typeof(IHostedService) && s.ImplementationType == typeof(ScheduledJobService))) - services.AddTransient(); - - if (!services.Any(s => s.ServiceType == typeof(JobManager) && s.ImplementationType == typeof(JobManager))) - services.AddSingleton(); - - if (!services.Any(s => s.ServiceType == typeof(IJobManager) && s.ImplementationType == typeof(JobManager))) - services.AddSingleton(sp => sp.GetRequiredService()); - - return services; - } - - public static IServiceCollection AddJobLifetimeService(this IServiceCollection services) - { - services.AddSingleton(); - services.AddSingleton(x => x.GetRequiredService()); return services; } } diff --git a/src/Foundatio.Extensions.Hosting/Jobs/JobManager.cs b/src/Foundatio.Extensions.Hosting/Jobs/JobManager.cs index d84fe2d8d..ebf8daa5f 100644 --- a/src/Foundatio.Extensions.Hosting/Jobs/JobManager.cs +++ b/src/Foundatio.Extensions.Hosting/Jobs/JobManager.cs @@ -5,12 +5,12 @@ using System.Threading; using System.Threading.Tasks; using Foundatio.Caching; -using Foundatio.Jobs; +using Foundatio.Jobs.Legacy; using Foundatio.Utility; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging; -namespace Foundatio.Extensions.Hosting.Jobs; +namespace Foundatio.Extensions.Hosting.Jobs.Legacy; public interface IJobManager { diff --git a/src/Foundatio.Extensions.Hosting/Jobs/JobOptionsBuilder.cs b/src/Foundatio.Extensions.Hosting/Jobs/JobOptionsBuilder.cs index 9adb13a94..0e9d13bfd 100644 --- a/src/Foundatio.Extensions.Hosting/Jobs/JobOptionsBuilder.cs +++ b/src/Foundatio.Extensions.Hosting/Jobs/JobOptionsBuilder.cs @@ -1,7 +1,7 @@ using System; -using Foundatio.Jobs; +using Foundatio.Jobs.Legacy; -namespace Foundatio.Extensions.Hosting.Jobs; +namespace Foundatio.Extensions.Hosting.Jobs.Legacy; public class HostedJobOptionsBuilder { diff --git a/src/Foundatio.Extensions.Hosting/Jobs/JobRuntimeService.cs b/src/Foundatio.Extensions.Hosting/Jobs/JobRuntimeService.cs new file mode 100644 index 000000000..8237ff0df --- /dev/null +++ b/src/Foundatio.Extensions.Hosting/Jobs/JobRuntimeService.cs @@ -0,0 +1,112 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Utility; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; + +namespace Foundatio.Extensions.Hosting.Jobs; + +/// +/// Options controlling the cadence and batch size of . +/// +public class JobRuntimeServiceOptions +{ + /// Whether the runtime pump runs. Default true; set false to take manual control of pumping. + public bool Enabled { get; set; } = true; + + /// + /// How often the runtime pump materializes CRON occurrences, dispatches due work, and runs queued jobs. + /// Defaults to one second so sub-minute CRON schedules and short delays are honored. + /// + public TimeSpan PollInterval { get; set; } = TimeSpan.FromSeconds(1); + + /// + /// Maximum number of due dispatches and queued jobs claimed per pump iteration. + /// + public int BatchSize { get; set; } = 100; + + /// + /// Maximum number of processing attempts for a durable job before a stale (lease-expired) instance is + /// dead-lettered instead of re-queued. Defaults to 3. + /// + public int MaxJobAttempts { get; set; } = 3; +} + +/// +/// Drives the durable job runtime introduced by . Without this hosted service nothing +/// materializes CRON occurrences, dispatches delayed/scheduled work, recovers stale (lease-expired) occurrences, or +/// runs jobs submitted through — the runtime store would accumulate work that never executes. +/// +public class JobRuntimeService : BackgroundService +{ + private readonly JobScheduleProcessor _processor; + private readonly IJobWorker _worker; + private readonly TimeProvider _timeProvider; + private readonly ILogger _logger; + private readonly JobRuntimeServiceOptions _options; + + public JobRuntimeService(JobScheduleProcessor processor, IJobWorker worker, TimeProvider? timeProvider = null, ILoggerFactory? loggerFactory = null, JobRuntimeServiceOptions? options = null) + { + _processor = processor ?? throw new ArgumentNullException(nameof(processor)); + _worker = worker ?? throw new ArgumentNullException(nameof(worker)); + _timeProvider = timeProvider ?? TimeProvider.System; + _logger = (loggerFactory ?? NullLoggerFactory.Instance).CreateLogger(); + _options = options ?? new JobRuntimeServiceOptions(); + } + + protected override async Task ExecuteAsync(CancellationToken stoppingToken) + { + if (!_options.Enabled) + { + _logger.LogInformation("Job runtime pump disabled (Enabled = false); not pumping the runtime store"); + return; + } + + _logger.LogInformation("Job runtime pump starting (poll interval {PollInterval}, batch size {BatchSize})", _options.PollInterval, _options.BatchSize); + + while (!stoppingToken.IsCancellationRequested) + { + try + { + var now = _timeProvider.GetUtcNow(); + + // Materialize CRON occurrences due within the misfire window (deduped, idempotent). + await _processor.EnqueueDueOccurrencesAsync(now, stoppingToken).AnyContext(); + + // Claim and run due dispatches: CRON occurrences plus delayed queue/pub-sub messages. This also + // recovers occurrences whose processing lease expired (crash mid-run) and applies retry/dead-letter. + await _processor.RunDueOccurrencesAsync(now, _options.BatchSize, lease: null, stoppingToken).AnyContext(); + + // Recover plain (non-CRON) jobs whose processing lease expired (a worker crash mid-run): re-queue them + // while attempts remain, otherwise dead-letter them. Without this they would strand in Processing. + await _worker.RecoverStaleAsync(_options.MaxJobAttempts, _options.BatchSize, stoppingToken).AnyContext(); + + // Run jobs submitted via IJobClient that are sitting in the Queued state. + await _worker.RunQueuedAsync(_options.BatchSize, stoppingToken).AnyContext(); + } + catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) + { + break; + } + catch (Exception ex) + { + _logger.LogError(ex, "Error pumping job runtime: {Message}", ex.Message); + } + + try + { + await _timeProvider.Delay(_options.PollInterval, stoppingToken).AnyContext(); + } + catch (OperationCanceledException) + { + break; + } + } + + _logger.LogInformation("Job runtime pump stopped"); + } +} diff --git a/src/Foundatio.Extensions.Hosting/Jobs/LegacyJobHostExtensions.cs b/src/Foundatio.Extensions.Hosting/Jobs/LegacyJobHostExtensions.cs new file mode 100644 index 000000000..ca428697a --- /dev/null +++ b/src/Foundatio.Extensions.Hosting/Jobs/LegacyJobHostExtensions.cs @@ -0,0 +1,216 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs.Legacy; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; + +namespace Foundatio.Extensions.Hosting.Jobs.Legacy; + +public static class LegacyJobHostExtensions +{ + public static IServiceCollection AddJob(this IServiceCollection services, HostedJobOptions jobOptions) + { + if (jobOptions.JobFactory == null) + throw new ArgumentNullException(nameof(jobOptions), "jobOptions.JobFactory is required"); + + return services.AddTransient(s => new HostedJobService(s, jobOptions, s.GetRequiredService())); + } + + public static IServiceCollection AddJob(this IServiceCollection services, HostedJobOptions? jobOptions = null) where T : class, IJob + { + services.AddTransient(); + return services.AddTransient(s => + { + if (jobOptions == null) + { + jobOptions = new HostedJobOptions(); + jobOptions.ApplyDefaults(); + } + + jobOptions.Name ??= JobOptions.GetDefaultJobName(typeof(T)); + jobOptions.JobFactory ??= sp => sp.GetRequiredService(); + + return new HostedJobService(s, jobOptions, s.GetRequiredService()); + }); + } + + public static IServiceCollection AddJob(this IServiceCollection services, Action configureJobOptions) where T : class, IJob + { + var jobOptionsBuilder = new HostedJobOptionsBuilder(); + jobOptionsBuilder.ApplyDefaults(); + jobOptionsBuilder.Name(JobOptions.GetDefaultJobName(typeof(T))); + configureJobOptions?.Invoke(jobOptionsBuilder); + return services.AddJob(jobOptionsBuilder.Target); + } + + public static IServiceCollection AddJob(this IServiceCollection services, Action configureJobOptions) + { + var jobOptionsBuilder = new HostedJobOptionsBuilder(); + configureJobOptions?.Invoke(jobOptionsBuilder); + return services.AddJob(jobOptionsBuilder.Target); + } + + public static IServiceCollection AddJob(this IServiceCollection services, string name, Func jobFactory, Action configureJobOptions) + { + var jobOptionsBuilder = new HostedJobOptionsBuilder(); + jobOptionsBuilder.Name(name).JobFactory(jobFactory); + configureJobOptions?.Invoke(jobOptionsBuilder); + return services.AddJob(jobOptionsBuilder.Target); + } + + /// + /// Legacy/compat. This registers the in-process , which runs CRON occurrences + /// in-process and does not materialize durable, recoverable occurrences. The forward path in the redesigned runtime + /// is the durable scheduler — register it with services.AddFoundatio().Jobs.UseInMemoryRuntime() plus + /// , which materializes durable + /// occurrences with retry, recovery, and dead-lettering. Routing this default API onto the durable scheduler is a + /// planned follow-up. + /// + public static IServiceCollection AddCronJob(this IServiceCollection services, ScheduledJobOptions jobOptions) + { + if (jobOptions.JobFactory == null) + throw new ArgumentNullException(nameof(jobOptions), "jobOptions.JobFactory is required"); + + services.AddJobScheduler(); + + return services.AddTransient(s => new ScheduledJobRegistration(jobOptions)); + } + + public static IServiceCollection AddCronJob(this IServiceCollection services, Action configureJobOptions) + { + var jobOptionsBuilder = new ScheduledJobOptionsBuilder(); + configureJobOptions?.Invoke(jobOptionsBuilder); + return services.AddCronJob(jobOptionsBuilder.Target); + } + + public static IServiceCollection AddCronJob(this IServiceCollection services, string cronSchedule, Action? configureJobOptions = null) where T : class, IJob + { + services.AddTransient(); + var jobOptionsBuilder = new ScheduledJobOptionsBuilder(); + jobOptionsBuilder.Name(JobOptions.GetDefaultJobName(typeof(T))).CronSchedule(cronSchedule).JobFactory(sp => sp.GetRequiredService()); + configureJobOptions?.Invoke(jobOptionsBuilder); + return services.AddCronJob(jobOptionsBuilder.Target); + } + + public static IServiceCollection AddCronJob(this IServiceCollection services, string name, string cronSchedule, Func action) + { + return services.AddCronJob(o => o.Name(name).CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, action))); + } + + public static IServiceCollection AddCronJob(this IServiceCollection services, string name, string cronSchedule, Func action) + { + return services.AddCronJob(o => o.Name(name).CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (xp, _) => action(xp)))); + } + + public static IServiceCollection AddCronJob(this IServiceCollection services, string name, string cronSchedule, Func action) + { + return services.AddCronJob(o => o.Name(name).CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (_, _) => action()))); + } + + public static IServiceCollection AddCronJob(this IServiceCollection services, string name, string cronSchedule, Action action) + { + return services.AddCronJob(o => o.Name(name).CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (xp, ct) => + { + action(xp, ct); + return Task.CompletedTask; + }))); + } + + public static IServiceCollection AddCronJob(this IServiceCollection services, string name, string cronSchedule, Action action) + { + return services.AddCronJob(o => o.Name(name).CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (_, ct) => + { + action(ct); + return Task.CompletedTask; + }))); + } + + public static IServiceCollection AddCronJob(this IServiceCollection services, string name, string cronSchedule, Action action) + { + return services.AddCronJob(o => o.Name(name).CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (_, _) => + { + action(); + return Task.CompletedTask; + }))); + } + + public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string cronSchedule, Action? configureJobOptions = null) where T : class, IJob + { + services.AddTransient(); + var jobOptionsBuilder = new ScheduledJobOptionsBuilder(); + jobOptionsBuilder.Name(JobOptions.GetDefaultJobName(typeof(T))).Distributed().CronSchedule(cronSchedule).JobFactory(sp => sp.GetRequiredService()); + configureJobOptions?.Invoke(jobOptionsBuilder); + return services.AddCronJob(jobOptionsBuilder.Target); + } + + public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string name, string cronSchedule, Func action) + { + return services.AddCronJob(o => o.Name(name).Distributed().CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, action))); + } + + public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string name, string cronSchedule, Func action) + { + return services.AddCronJob(o => o.Name(name).Distributed().CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (xp, _) => action(xp)))); + } + + public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string name, string cronSchedule, Func action) + { + return services.AddCronJob(o => o.Name(name).Distributed().CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (_, _) => action()))); + } + + public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string name, string cronSchedule, Action action) + { + return services.AddCronJob(o => o.Name(name).Distributed().CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (xp, ct) => + { + action(xp, ct); + return Task.CompletedTask; + }))); + } + + public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string name, string cronSchedule, Action action) + { + return services.AddCronJob(o => o.Name(name).Distributed().CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (_, ct) => + { + action(ct); + return Task.CompletedTask; + }))); + } + + public static IServiceCollection AddDistributedCronJob(this IServiceCollection services, string name, string cronSchedule, Action action) + { + return services.AddCronJob(o => o.Name(name).Distributed().CronSchedule(cronSchedule).JobFactory(sp => new DynamicJob(sp, (_, _) => + { + action(); + return Task.CompletedTask; + }))); + } + + /// + /// Legacy/compat: registers the in-process CRON scheduler. For durable, + /// recoverable CRON occurrences use the redesigned runtime (AddFoundatio().Jobs.UseInMemoryRuntime() + + /// ) instead. + /// + public static IServiceCollection AddJobScheduler(this IServiceCollection services) + { + if (!services.Any(s => s.ServiceType == typeof(IHostedService) && s.ImplementationType == typeof(ScheduledJobService))) + services.AddTransient(); + + if (!services.Any(s => s.ServiceType == typeof(JobManager) && s.ImplementationType == typeof(JobManager))) + services.AddSingleton(); + + if (!services.Any(s => s.ServiceType == typeof(IJobManager) && s.ImplementationType == typeof(JobManager))) + services.AddSingleton(sp => sp.GetRequiredService()); + + return services; + } + + public static IServiceCollection AddJobLifetimeService(this IServiceCollection services) + { + services.AddSingleton(); + services.AddSingleton(x => x.GetRequiredService()); + return services; + } +} diff --git a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobInstance.cs b/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobInstance.cs index 8fe31d801..de9b59697 100644 --- a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobInstance.cs +++ b/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobInstance.cs @@ -5,16 +5,16 @@ using System.Threading; using System.Threading.Tasks; using Foundatio.Caching; -using Foundatio.Extensions.Hosting.Cronos; -using Foundatio.Jobs; +using Foundatio.Cronos; +using Foundatio.Jobs.Legacy; using Foundatio.Lock; -using Foundatio.Messaging; +using Foundatio.Messaging.Legacy; using Foundatio.Utility; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; -namespace Foundatio.Extensions.Hosting.Jobs; +namespace Foundatio.Extensions.Hosting.Jobs.Legacy; internal class ScheduledJobInstance { diff --git a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobOptions.cs b/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobOptions.cs index b1279a284..95559f577 100644 --- a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobOptions.cs +++ b/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobOptions.cs @@ -1,9 +1,9 @@ using System; using System.ComponentModel; using System.Runtime.CompilerServices; -using Foundatio.Jobs; +using Foundatio.Jobs.Legacy; -namespace Foundatio.Extensions.Hosting.Jobs; +namespace Foundatio.Extensions.Hosting.Jobs.Legacy; public class ScheduledJobOptions : INotifyPropertyChanged { diff --git a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobOptionsBuilder.cs b/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobOptionsBuilder.cs index 5fda06683..2b104f27e 100644 --- a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobOptionsBuilder.cs +++ b/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobOptionsBuilder.cs @@ -1,9 +1,9 @@ using System; using System.Threading; using System.Threading.Tasks; -using Foundatio.Jobs; +using Foundatio.Jobs.Legacy; -namespace Foundatio.Extensions.Hosting.Jobs; +namespace Foundatio.Extensions.Hosting.Jobs.Legacy; public class ScheduledJobOptionsBuilder { diff --git a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobRegistration.cs b/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobRegistration.cs index 6d7269e75..862a0889c 100644 --- a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobRegistration.cs +++ b/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobRegistration.cs @@ -1,4 +1,4 @@ -namespace Foundatio.Extensions.Hosting.Jobs; +namespace Foundatio.Extensions.Hosting.Jobs.Legacy; public class ScheduledJobRegistration { diff --git a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobService.cs b/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobService.cs index 39bcdfbd9..b557099c2 100644 --- a/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobService.cs +++ b/src/Foundatio.Extensions.Hosting/Jobs/ScheduledJobService.cs @@ -4,15 +4,20 @@ using System.Threading.Tasks; using Foundatio.Caching; using Foundatio.Extensions.Hosting.Startup; -using Foundatio.Messaging; +using Foundatio.Messaging.Legacy; using Foundatio.Utility; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; -namespace Foundatio.Extensions.Hosting.Jobs; +namespace Foundatio.Extensions.Hosting.Jobs.Legacy; +/// +/// Legacy/compat in-process CRON scheduler used by . +/// It runs occurrences in-process and does not materialize durable, recoverable occurrences. The redesigned runtime's +/// durable scheduler (JobScheduleProcessor driven by ) is the forward path. +/// public class ScheduledJobService : BackgroundService { private readonly IServiceProvider _serviceProvider; diff --git a/src/Foundatio.Extensions.Hosting/Jobs/ShutdownHostIfNoJobsRunningService.cs b/src/Foundatio.Extensions.Hosting/Jobs/ShutdownHostIfNoJobsRunningService.cs index bb5a6c74d..1dd387ad9 100644 --- a/src/Foundatio.Extensions.Hosting/Jobs/ShutdownHostIfNoJobsRunningService.cs +++ b/src/Foundatio.Extensions.Hosting/Jobs/ShutdownHostIfNoJobsRunningService.cs @@ -10,7 +10,7 @@ using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; -namespace Foundatio.Extensions.Hosting.Jobs; +namespace Foundatio.Extensions.Hosting.Jobs.Legacy; public class ShutdownHostIfNoJobsRunningService : IHostedService, IDisposable { diff --git a/src/Foundatio.Redis/Foundatio.Redis.csproj b/src/Foundatio.Redis/Foundatio.Redis.csproj new file mode 100644 index 000000000..f70dd8073 --- /dev/null +++ b/src/Foundatio.Redis/Foundatio.Redis.csproj @@ -0,0 +1,11 @@ + + + Redis-backed durable job runtime store (IJobRuntimeStore) for Foundatio. Temporary in-repo provider for validating the redesigned job runtime against a real distributed store. + + + + + + + + diff --git a/src/Foundatio.Redis/Messaging/RedisStreamsMessageTransport.cs b/src/Foundatio.Redis/Messaging/RedisStreamsMessageTransport.cs new file mode 100644 index 000000000..72df233b5 --- /dev/null +++ b/src/Foundatio.Redis/Messaging/RedisStreamsMessageTransport.cs @@ -0,0 +1,511 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; +using StackExchange.Redis; + +namespace Foundatio.Messaging; + +/// +/// An over Redis Streams + consumer groups (at-least-once). Temporary in-repo provider +/// used to validate the redesigned transport contract — and the core's retry/dead-letter machinery — against a real +/// broker. A stream is a queue/topic; a consumer group is a subscription (the default group for a plain queue gives +/// competing consumers; one group per named subscription gives topic fan-out). +/// +/// +/// Streams has no per-message visible-until or per-message delay, so this transport keeps the lease explicitly: a +/// per-group sorted set (member = stream entry id, score = visible-until unix-ms) is the authoritative +/// in-flight lease and a per-group hash holds token|delivery-count per entry. Reclaim (abandon, redelivery +/// delay, lock expiry, crashed consumer) is driven by that sorted set — entries whose lease has lapsed are +/// XCLAIMed and redelivered (same stream id, delivery count incremented). Because the lease lives in Redis, a +/// message held by a crashed instance is recovered by any other instance. A stale receipt (already settled, or the +/// entry was redelivered to someone else) is detected by an owner token and surfaced as . +/// +public sealed class RedisStreamsMessageTransport : IMessageTransport, ISupportsPull, ISupportsVisibilityTimeout, + ISupportsLockRenewal, ISupportsRedeliveryDelay, ISupportsDeadLetter, ISupportsProvisioning, ISupportsStats, ITransportInfo +{ + private static readonly TimeSpan PollInterval = TimeSpan.FromMilliseconds(250); + + private static readonly IReadOnlySet _supportedRoles = + new HashSet { DestinationRole.Queue, DestinationRole.Topic, DestinationRole.Subscription, DestinationRole.Binding }; + + private readonly RedisStreamsMessageTransportOptions _options; + private readonly IDatabase _db; + private readonly TimeProvider _timeProvider; + private readonly string _prefix; + private readonly string _consumer; + private readonly ConcurrentDictionary _ensuredGroups = new(StringComparer.Ordinal); + private int _isDisposed; + + public RedisStreamsMessageTransport(RedisStreamsMessageTransportOptions options) + { + _options = options ?? throw new ArgumentNullException(nameof(options)); + ArgumentNullException.ThrowIfNull(options.ConnectionMultiplexer); + _db = options.ConnectionMultiplexer.GetDatabase(); + _timeProvider = options.TimeProvider ?? TimeProvider.System; + _prefix = options.KeyPrefix ?? ""; + _consumer = !String.IsNullOrEmpty(options.ConsumerName) ? options.ConsumerName : $"c-{Guid.NewGuid():N}"[..16]; + } + + // Streams append FIFO; there is no native priority, per-message expiration, or delayed delivery (delays route + // through the runtime-store fallback), and no broker-imposed size or batch limits. + private static readonly TransportCapabilities _capabilities = new() { Ordering = OrderingGuarantee.Fifo }; + + public DeliveryGuarantee DeliveryGuarantee => DeliveryGuarantee.AtLeastOnce; + public IReadOnlySet SupportedRoles => _supportedRoles; + public TransportCapabilities GetCapabilities(DestinationRole role) => _capabilities; + public TimeSpan? MaxRedeliveryDelay => null; // lease is tracked in Redis, so any delay is honored + public TimeSpan? MaxVisibilityTimeout => null; + + public async Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(destination); + ArgumentNullException.ThrowIfNull(messages); + + // Streams have no native delayed delivery; the core routes delayed sends through the runtime-store fallback + // (no DelayedDelivery capability is advertised), so a future DeliverAt reaching here is a contract violation — + // refuse loudly rather than deliver immediately and silently drop the delay. + if (options.DeliverAt is { } deliverAt && deliverAt > _timeProvider.GetUtcNow()) + throw new NotSupportedException($"Transport \"{nameof(RedisStreamsMessageTransport)}\" does not support native delayed delivery. Register a job runtime store so delayed sends use the scheduled-dispatch fallback."); + + // The stream IS the queue/topic; subscriptions read it through their own group. The address role picks the + // stream namespace so a queue and a topic sharing a route name never cross-deliver. + RedisKey streamKey = destination.Role == DestinationRole.Topic ? TopicStreamKey(destination.Name) : QueueStreamKey(destination.Name); + var items = new List(messages.Count); + foreach (var message in messages) + { + RedisValue id = await _db.StreamAddAsync(streamKey, BuildFields(message), messageId: null, + maxLength: _options.MaxStreamLength, useApproximateMaxLength: true).ConfigureAwait(false); + items.Add(new SendItemResult { MessageId = id.ToString() }); + } + + return new SendResult { Items = items }; + } + + public Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct) + => ReceiveAsync(source, request, _options.DefaultVisibilityTimeout, ct); + + public async Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, TimeSpan visibility, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(source); + ArgumentNullException.ThrowIfNull(request); + + var resolved = Resolve(source); + await EnsureGroupAsync(resolved).ConfigureAwait(false); + + int max = Math.Max(1, request.MaxMessages); + long visibilityMs = (long)Math.Max(0, visibility.TotalMilliseconds); + var deadline = _timeProvider.GetUtcNow() + (request.MaxWaitTime ?? TimeSpan.Zero); + + while (true) + { + ct.ThrowIfCancellationRequested(); + var entries = await PollOnceAsync(source, resolved, max, visibilityMs, ct).ConfigureAwait(false); + if (entries.Count > 0) + return entries; + + var remaining = deadline - _timeProvider.GetUtcNow(); + if (remaining <= TimeSpan.Zero) + return []; + + await Task.Delay(remaining < PollInterval ? remaining : PollInterval, ct).ConfigureAwait(false); + } + } + + private async Task> PollOnceAsync(DestinationAddress source, ResolvedSource resolved, int max, long visibilityMs, CancellationToken ct) + { + var result = new List(max); + long nowMs = _timeProvider.GetUtcNow().ToUnixTimeMilliseconds(); + RedisKey lockKey = LockKey(resolved); + RedisKey metaKey = MetaKey(resolved); + + // 1. Reclaim entries whose lease has lapsed (abandoned, redelivery-delay due, lock expired, crashed consumer). + // The lease score is updated only after the claim, never removed first, so a crash mid-reclaim can't orphan an + // entry (it stays reclaimable); the cost is that two instances racing the same lapsed entry may both deliver it + // — acceptable under at-least-once. + var dueIds = await _db.SortedSetRangeByScoreAsync(lockKey, Double.NegativeInfinity, nowMs, take: max).ConfigureAwait(false); + if (dueIds.Length > 0) + { + var claimed = await _db.StreamClaimAsync(resolved.StreamKey, resolved.Group, _consumer, 0, dueIds).ConfigureAwait(false); + foreach (var entry in claimed) + { + ct.ThrowIfCancellationRequested(); + if (entry.IsNull || entry.Values is not { Length: > 0 }) + { + // The entry was settled/trimmed since we read the lease; drop our bookkeeping for it. + await _db.SortedSetRemoveAsync(lockKey, entry.Id).ConfigureAwait(false); + await _db.HashDeleteAsync(metaKey, entry.Id).ConfigureAwait(false); + continue; + } + + int deliveries = ParseDeliveries(await _db.HashGetAsync(metaKey, entry.Id).ConfigureAwait(false)) + 1; + result.Add(await TrackAsync(source, resolved, entry, deliveries, nowMs, visibilityMs).ConfigureAwait(false)); + if (result.Count >= max) + return result; + } + } + + // 2. New, never-delivered entries. + var fresh = await _db.StreamReadGroupAsync(resolved.StreamKey, resolved.Group, _consumer, StreamPosition.NewMessages, max - result.Count).ConfigureAwait(false); + foreach (var entry in fresh) + { + ct.ThrowIfCancellationRequested(); + result.Add(await TrackAsync(source, resolved, entry, 1, nowMs, visibilityMs).ConfigureAwait(false)); + } + + return result; + } + + // Records the lease (sorted set) + owner token & delivery count (hash) for a just-delivered entry and projects it + // into a TransportEntry whose Receipt carries everything needed to settle it. + private async Task TrackAsync(DestinationAddress source, ResolvedSource resolved, StreamEntry entry, int deliveries, long nowMs, long visibilityMs) + { + string token = Guid.NewGuid().ToString("N"); + await _db.HashSetAsync(MetaKey(resolved), entry.Id, $"{token}|{deliveries}").ConfigureAwait(false); + await _db.SortedSetAddAsync(LockKey(resolved), entry.Id, nowMs + visibilityMs).ConfigureAwait(false); + return ToEntry(source, resolved, entry, deliveries, token); + } + + public async Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) + { + ThrowIfDisposed(); + var r = await ValidateReceiptAsync(entry).ConfigureAwait(false); + + long acked = await _db.StreamAcknowledgeAsync(r.StreamKey, r.Group, r.EntryId).ConfigureAwait(false); + // Only a queue stream (single consumer group) may delete on complete. A topic stream is shared by every + // subscription group, so one group completing must not delete the entry before the others read it; topic + // entries are retained (bound by MaxStreamLength when configured). + if (!IsTopicStream(r.StreamKey)) + await _db.StreamDeleteAsync(r.StreamKey, [r.EntryId]).ConfigureAwait(false); + await ClearTrackingAsync(r).ConfigureAwait(false); + + if (acked == 0) + throw new ReceiptExpiredException(); + } + + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) => AbandonAsync(entry, TimeSpan.Zero, ct); + + public async Task AbandonAsync(TransportEntry entry, TimeSpan redeliveryDelay, CancellationToken ct) + { + ThrowIfDisposed(); + var r = await ValidateReceiptAsync(entry).ConfigureAwait(false); + + // Make the (still-pending) entry reclaimable when the delay lapses; the reclaim pass redelivers the same stream + // id with an incremented delivery count. delay <= 0 => immediately due. + long dueMs = _timeProvider.GetUtcNow().ToUnixTimeMilliseconds() + (long)Math.Max(0, redeliveryDelay.TotalMilliseconds); + await _db.SortedSetAddAsync(LockKey(r), r.EntryId, dueMs).ConfigureAwait(false); + } + + public async Task RenewLockAsync(TransportEntry entry, TimeSpan? duration, CancellationToken ct) + { + ThrowIfDisposed(); + var r = await ValidateReceiptAsync(entry).ConfigureAwait(false); + long until = _timeProvider.GetUtcNow().ToUnixTimeMilliseconds() + (long)(duration ?? _options.DefaultVisibilityTimeout).TotalMilliseconds; + await _db.SortedSetAddAsync(LockKey(r), r.EntryId, until).ConfigureAwait(false); + } + + public async Task DeadLetterAsync(TransportEntry entry, string? reason, CancellationToken ct) + { + ThrowIfDisposed(); + var r = await ValidateReceiptAsync(entry).ConfigureAwait(false); + + // Match the in-memory reference: record the reason header only when there's a reason (never an empty value). + var headerBuilder = entry.Headers.ToBuilder(); + if (!String.IsNullOrEmpty(reason)) + headerBuilder.Set(KnownHeaders.DeadLetterReason, reason); + var headers = headerBuilder.Build(); + await _db.StreamAddAsync(DeadKey(r.StreamKey), BuildFields(entry.Id, entry.Body, headers), messageId: null, + maxLength: _options.MaxStreamLength, useApproximateMaxLength: true).ConfigureAwait(false); + + await _db.StreamAcknowledgeAsync(r.StreamKey, r.Group, r.EntryId).ConfigureAwait(false); + // Same rule as CompleteAsync: other subscription groups on a topic stream may not have read this entry yet. + if (!IsTopicStream(r.StreamKey)) + await _db.StreamDeleteAsync(r.StreamKey, [r.EntryId]).ConfigureAwait(false); + await ClearTrackingAsync(r).ConfigureAwait(false); + } + + public async Task> ReceiveDeadLetteredAsync(DestinationAddress destination, ReceiveRequest request, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(destination); + ArgumentNullException.ThrowIfNull(request); + + RedisKey deadKey = DeadKey(Resolve(destination).StreamKey); + var entries = await _db.StreamRangeAsync(deadKey, count: Math.Max(1, request.MaxMessages)).ConfigureAwait(false); + if (entries.Length == 0) + return []; + + var result = new List(entries.Length); + var ids = new RedisValue[entries.Length]; + for (int i = 0; i < entries.Length; i++) + { + ids[i] = entries[i].Id; + result.Add(ToEntry(destination, resolved: null, entries[i], deliveries: 1, token: "")); + } + + // Inspecting the dead-letter backlog consumes it. + await _db.StreamDeleteAsync(deadKey, ids).ConfigureAwait(false); + return result; + } + + public async Task EnsureAsync(IReadOnlyList declarations, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(declarations); + + foreach (var declaration in declarations) + { + switch (declaration.Address.Role) + { + case DestinationRole.Topic: + // Topics are read through subscription groups; nothing to create until a subscription appears. + break; + default: + // Queue, subscription, and binding declarations all materialize as a consumer group on the stream + // the address resolves to. + await EnsureGroupAsync(Resolve(declaration.Address)).ConfigureAwait(false); + break; + } + } + } + + public async Task DeleteAsync(DestinationAddress destination, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(destination); + + // A subscription deletes only that group's state (never the shared topic stream — other subscriptions still + // read it); a queue or topic deletes its own stream and everything scoped to it. + if (destination.Role is DestinationRole.Subscription or DestinationRole.Binding) + { + var sub = Resolve(destination); + await _db.StreamDeleteConsumerGroupAsync(sub.StreamKey, sub.Group).ConfigureAwait(false); + await _db.KeyDeleteAsync([LockKey(sub), MetaKey(sub)]).ConfigureAwait(false); + _ensuredGroups.TryRemove(GroupKey(sub), out _); + return; + } + + var resolved = Resolve(destination); + + // Drop each consumer group's lease/meta state before the stream itself (topic streams can carry several). + if (await _db.KeyExistsAsync(resolved.StreamKey).ConfigureAwait(false)) + { + foreach (var group in await _db.StreamGroupInfoAsync(resolved.StreamKey).ConfigureAwait(false)) + { + var groupSource = resolved with { Group = group.Name }; + await _db.KeyDeleteAsync([LockKey(groupSource), MetaKey(groupSource)]).ConfigureAwait(false); + _ensuredGroups.TryRemove(GroupKey(groupSource), out _); + } + } + + await _db.KeyDeleteAsync([resolved.StreamKey, DeadKey(resolved.StreamKey)]).ConfigureAwait(false); + _ensuredGroups.TryRemove(GroupKey(resolved), out _); + } + + public async Task ExistsAsync(DestinationAddress destination, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(destination); + + var resolved = Resolve(destination); + if (!await _db.KeyExistsAsync(resolved.StreamKey).ConfigureAwait(false)) + return false; + + // A subscription exists when its consumer group exists on the topic stream; a queue/topic exists when its + // stream key does. + if (destination.Role is not (DestinationRole.Subscription or DestinationRole.Binding)) + return true; + + foreach (var group in await _db.StreamGroupInfoAsync(resolved.StreamKey).ConfigureAwait(false)) + { + if (String.Equals(group.Name, resolved.Group, StringComparison.Ordinal)) + return true; + } + + return false; + } + + public async Task GetStatsAsync(DestinationAddress destination, CancellationToken ct) + { + ThrowIfDisposed(); + ArgumentNullException.ThrowIfNull(destination); + var resolved = Resolve(destination); + + // Probing stats must not create phantom streams/groups; a destination that doesn't exist yet is simply empty. + if (!await _db.KeyExistsAsync(resolved.StreamKey).ConfigureAwait(false)) + return new MessageDestinationStats(); + + await EnsureGroupAsync(resolved).ConfigureAwait(false); + + long length = await _db.StreamLengthAsync(resolved.StreamKey).ConfigureAwait(false); + long working = (await _db.StreamPendingAsync(resolved.StreamKey, resolved.Group).ConfigureAwait(false)).PendingMessageCount; + RedisKey deadKey = DeadKey(resolved.StreamKey); + long dead = await _db.KeyExistsAsync(deadKey).ConfigureAwait(false) ? await _db.StreamLengthAsync(deadKey).ConfigureAwait(false) : 0; + + return new MessageDestinationStats + { + Queued = Math.Max(0, length - working), + Working = working, + Deadletter = dead + }; + } + + public ValueTask DisposeAsync() + { + Interlocked.Exchange(ref _isDisposed, 1); + return ValueTask.CompletedTask; // the connection multiplexer is owned by the caller + } + + private async Task ValidateReceiptAsync(TransportEntry entry) + { + if (entry.Receipt.TransportState is not StreamReceipt r) + throw new ReceiptExpiredException("The transport entry does not carry a Redis Streams receipt."); + + // The owner token guards stale receipts: once the entry is redelivered (reclaimed) or settled, the token in the + // meta hash no longer matches, so a late Complete/Abandon from the previous holder is rejected. + var current = await _db.HashGetAsync(MetaKey(r), r.EntryId).ConfigureAwait(false); + if (current.IsNull || ParseToken(current) != r.Token) + throw new ReceiptExpiredException(); + + return r; + } + + private async Task ClearTrackingAsync(StreamReceipt r) + { + await _db.SortedSetRemoveAsync(LockKey(r), r.EntryId).ConfigureAwait(false); + await _db.HashDeleteAsync(MetaKey(r), r.EntryId).ConfigureAwait(false); + } + + private async Task EnsureGroupAsync(ResolvedSource resolved) + { + if (!_ensuredGroups.TryAdd(GroupKey(resolved), 0)) + return; + + try + { + await _db.StreamCreateConsumerGroupAsync(resolved.StreamKey, resolved.Group, resolved.Position, createStream: true).ConfigureAwait(false); + } + catch (RedisServerException ex) when (ex.Message.Contains("BUSYGROUP", StringComparison.Ordinal)) + { + // Group already exists — creation is idempotent. + } + } + + // The address is structural, so the physical mapping is derived from it directly — topology declarations and the + // runtime resolve the SAME address to the SAME stream/group, with no registration cache to drift. A subscription is + // a consumer group on its owning topic's stream: the group is named by the bare subscription name and scoped by the + // topic stream key. A queue is its own stream read through the shared default group (competing consumers). + private ResolvedSource Resolve(DestinationAddress address) => address.Role switch + { + DestinationRole.Topic => new ResolvedSource(TopicStreamKey(address.Name), _options.DefaultConsumerGroup, "$"), + DestinationRole.Subscription or DestinationRole.Binding => new ResolvedSource(TopicStreamKey(address.Topic ?? address.Name), address.Name, "$"), + _ => new ResolvedSource(QueueStreamKey(address.Name), _options.DefaultConsumerGroup, "0") + }; + + private TransportEntry ToEntry(DestinationAddress destination, ResolvedSource? resolved, StreamEntry entry, int deliveries, string token) + { + string? messageId = GetField(entry, "id"); + var headers = MessageHeaders.DeserializeFromJson(GetField(entry, "h")); + Receipt receipt = resolved is null + ? default + : new Receipt { TransportState = new StreamReceipt(resolved.StreamKey.ToString(), resolved.Group, entry.Id.ToString(), token) }; + + return new TransportEntry + { + Id = String.IsNullOrEmpty(messageId) ? entry.Id.ToString() : messageId, + Destination = destination, + Body = GetBody(entry), + Headers = headers, + DeliveryCount = deliveries, + EnqueuedUtc = ParseStreamIdTime(entry.Id), + Receipt = receipt + }; + } + + private static NameValueEntry[] BuildFields(TransportMessage message) + => BuildFields(message.MessageId, message.Body, message.Headers, message.ContentType); + + private static NameValueEntry[] BuildFields(string? messageId, ReadOnlyMemory body, MessageHeaders headers, string? contentType = null) + { + return + [ + new NameValueEntry("id", messageId ?? ""), + new NameValueEntry("ct", contentType ?? ""), + new NameValueEntry("h", MessageHeaders.SerializeToJson(headers)), + new NameValueEntry("b", body.ToArray()) + ]; + } + + private static string? GetField(StreamEntry entry, string name) + { + foreach (var value in entry.Values) + { + if (value.Name == name) + return value.Value.IsNull ? null : value.Value.ToString(); + } + + return null; + } + + private static ReadOnlyMemory GetBody(StreamEntry entry) + { + foreach (var value in entry.Values) + { + if (value.Name == "b") + return value.Value.IsNullOrEmpty ? ReadOnlyMemory.Empty : (byte[])value.Value!; + } + + return ReadOnlyMemory.Empty; + } + + + // Stream ids are "-"; the timestamp half is the broker enqueue time. + private static DateTimeOffset? ParseStreamIdTime(RedisValue id) + { + string s = id.ToString(); + int dash = s.IndexOf('-'); + string ms = dash > 0 ? s[..dash] : s; + return Int64.TryParse(ms, NumberStyles.Integer, CultureInfo.InvariantCulture, out long unixMs) + ? DateTimeOffset.FromUnixTimeMilliseconds(unixMs) + : null; + } + + private static int ParseDeliveries(RedisValue meta) + { + if (meta.IsNullOrEmpty) + return 0; + string s = meta.ToString(); + int bar = s.IndexOf('|'); + return bar >= 0 && Int32.TryParse(s.AsSpan(bar + 1), NumberStyles.Integer, CultureInfo.InvariantCulture, out int n) ? n : 0; + } + + private static string ParseToken(RedisValue meta) + { + string s = meta.ToString(); + int bar = s.IndexOf('|'); + return bar >= 0 ? s[..bar] : s; + } + + // Streams are namespaced by role ("q:" queue, "t:" topic) because an XADD lands on whichever stream the key names: + // without the split, a message type both sent and published would share one stream and cross-deliver (a publish + // consumed as queue work and vice versa). Subscriptions are consumer groups on the topic stream. + private RedisKey QueueStreamKey(string name) => $"{_prefix}q:{name}"; + private RedisKey TopicStreamKey(string name) => $"{_prefix}t:{name}"; + private bool IsTopicStream(string streamKey) => streamKey.StartsWith($"{_prefix}t:", StringComparison.Ordinal); + private static RedisKey DeadKey(RedisKey streamKey) => streamKey.ToString() + ":dead"; + private static RedisKey LockKey(ResolvedSource r) => $"{r.StreamKey}:lock:{r.Group}"; + private static RedisKey MetaKey(ResolvedSource r) => $"{r.StreamKey}:meta:{r.Group}"; + private static RedisKey LockKey(StreamReceipt r) => $"{r.StreamKey}:lock:{r.Group}"; + private static RedisKey MetaKey(StreamReceipt r) => $"{r.StreamKey}:meta:{r.Group}"; + private static string GroupKey(ResolvedSource r) => $"{r.StreamKey}|{r.Group}"; + + private void ThrowIfDisposed() => ObjectDisposedException.ThrowIf(Volatile.Read(ref _isDisposed) == 1, this); + + private sealed record ResolvedSource(RedisKey StreamKey, string Group, RedisValue Position); + + private sealed record StreamReceipt(string StreamKey, string Group, string EntryId, string Token); +} diff --git a/src/Foundatio.Redis/Messaging/RedisStreamsMessageTransportOptions.cs b/src/Foundatio.Redis/Messaging/RedisStreamsMessageTransportOptions.cs new file mode 100644 index 000000000..2d053e356 --- /dev/null +++ b/src/Foundatio.Redis/Messaging/RedisStreamsMessageTransportOptions.cs @@ -0,0 +1,28 @@ +using System; +using StackExchange.Redis; + +namespace Foundatio.Messaging; + +public class RedisStreamsMessageTransportOptions +{ + /// The Redis connection to use. Required. + public IConnectionMultiplexer ConnectionMultiplexer { get; set; } = null!; + + /// Prefix applied to every stream/key this transport creates. Isolates environments/runs on a shared Redis. + public string KeyPrefix { get; set; } = "fnd:msg:"; + + /// Consumer-group name used for plain queue destinations (its members are competing consumers). + public string DefaultConsumerGroup { get; set; } = "foundatio"; + + /// How long a received message stays invisible to other consumers before it can be reclaimed (the lease). + public TimeSpan DefaultVisibilityTimeout { get; set; } = TimeSpan.FromSeconds(30); + + /// Approximate MAXLEN cap applied on XADD (null = no trimming). Trimming can drop un-acked entries; keep ample headroom. + public int? MaxStreamLength { get; set; } + + /// This node's consumer name within every group (defaults to a stable per-instance id). Distinct instances are competing consumers. + public string? ConsumerName { get; set; } + + /// Time source (defaults to ). + public TimeProvider? TimeProvider { get; set; } +} diff --git a/src/Foundatio.Redis/RedisFoundatioBuilderExtensions.cs b/src/Foundatio.Redis/RedisFoundatioBuilderExtensions.cs new file mode 100644 index 000000000..db7dab9f4 --- /dev/null +++ b/src/Foundatio.Redis/RedisFoundatioBuilderExtensions.cs @@ -0,0 +1,55 @@ +using System; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using StackExchange.Redis; + +namespace Foundatio; + +public static class RedisFoundatioBuilderExtensions +{ + /// + /// Backs the durable job runtime with Redis. Uses an already registered in DI, + /// otherwise connects using or the "Redis" connection string from configuration + /// (falling back to localhost). When both messaging and jobs use Redis a single connection is shared, so the + /// connection string from the first UseRedis call wins (a differing string on the second call is ignored). + /// + public static FoundatioBuilder UseRedis(this FoundatioBuilder.JobsBuilder builder, Action? configure = null, string? connectionString = null) + { + EnsureConnection(((IFoundatioBuilder)builder).Services, connectionString); + return builder.UseRuntimeStore(sp => + { + var options = new RedisJobRuntimeStoreOptions { ConnectionMultiplexer = sp.GetRequiredService() }; + configure?.Invoke(options); + return new RedisJobRuntimeStore(options); + }); + } + + /// + /// Runs messaging (queues + pub/sub) over Redis Streams. Uses an already + /// registered in DI, otherwise connects using or the "Redis" connection string + /// from configuration (falling back to localhost). When both messaging and jobs use Redis a single connection is + /// shared, so the connection string from the first UseRedis call wins (a differing string on the second is ignored). + /// + public static FoundatioBuilder UseRedis(this FoundatioBuilder.MessagingBuilder builder, Action? configure = null, string? connectionString = null) + { + EnsureConnection(((IFoundatioBuilder)builder).Services, connectionString); + return builder.UseTransport(sp => + { + var options = new RedisStreamsMessageTransportOptions { ConnectionMultiplexer = sp.GetRequiredService() }; + configure?.Invoke(options); + return new RedisStreamsMessageTransport(options); + }); + } + + // Register a single shared multiplexer if the app hasn't already, so messaging and jobs reuse one connection. + private static void EnsureConnection(IServiceCollection services, string? connectionString) + { + services.TryAddSingleton(sp => ConnectionMultiplexer.Connect( + connectionString + ?? sp.GetService()?.GetConnectionString("Redis") + ?? "localhost:6379")); + } +} diff --git a/src/Foundatio.Redis/RedisJobRuntimeStore.cs b/src/Foundatio.Redis/RedisJobRuntimeStore.cs new file mode 100644 index 000000000..ea88cb1a3 --- /dev/null +++ b/src/Foundatio.Redis/RedisJobRuntimeStore.cs @@ -0,0 +1,508 @@ +using System; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Text.Json; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using StackExchange.Redis; + +namespace Foundatio.Jobs; + +/// +/// A Redis-backed . Temporary in-repo provider used to validate the durable job runtime +/// (state transitions, leases/claims, scheduled dispatches) against a real distributed store. +/// +/// +/// Job state is a hash at {prefix}job:{id}; status and name indexes are sets; due dispatches are a sorted set +/// scored by due time. Conditional transitions use Redis transactions with hash-field conditions (optimistic +/// concurrency), so a state change only commits if the fields it was predicated on are unchanged — including a +/// lease-value condition that makes reclaim safe against a concurrent renew. Times are stored as UTC ticks for +/// unambiguous numeric comparison. +/// +public sealed class RedisJobRuntimeStore : IJobRuntimeStore +{ + private const string ClaimDueScript = """ + local ids = redis.call('ZRANGEBYSCORE', KEYS[1], '-inf', ARGV[1], 'LIMIT', 0, tonumber(ARGV[2])) + local claimed = {} + for _, id in ipairs(ids) do + local dkey = ARGV[5] .. id + if redis.call('EXISTS', dkey) == 1 then + local owner = redis.call('HGET', dkey, 'claimOwner') + local expires = redis.call('HGET', dkey, 'claimExpiresUtc') + if (not owner or owner == '') or (expires and expires ~= '' and tonumber(expires) <= tonumber(ARGV[1])) then + redis.call('HSET', dkey, 'claimOwner', ARGV[3], 'claimExpiresUtc', ARGV[4]) + redis.call('HINCRBY', dkey, 'attempts', 1) + table.insert(claimed, id) + end + end + end + return claimed + """; + + private readonly IDatabase _db; + private readonly string _prefix; + private readonly TimeProvider _timeProvider; + + public RedisJobRuntimeStore(RedisJobRuntimeStoreOptions options) + { + ArgumentNullException.ThrowIfNull(options); + ArgumentNullException.ThrowIfNull(options.ConnectionMultiplexer); + _db = options.ConnectionMultiplexer.GetDatabase(); + _prefix = options.KeyPrefix ?? ""; + _timeProvider = options.TimeProvider ?? TimeProvider.System; + } + + public RedisJobRuntimeStore(IConnectionMultiplexer connectionMultiplexer, string keyPrefix = "fnd:jobs:", TimeProvider? timeProvider = null) + : this(new RedisJobRuntimeStoreOptions { ConnectionMultiplexer = connectionMultiplexer, KeyPrefix = keyPrefix, TimeProvider = timeProvider }) { } + + public Task CreateIfAbsentAsync(JobState initial, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(initial); + cancellationToken.ThrowIfCancellationRequested(); + + var now = _timeProvider.GetUtcNow(); + var state = initial with + { + CreatedUtc = initial.CreatedUtc == default ? now : initial.CreatedUtc, + LastUpdatedUtc = initial.LastUpdatedUtc == default ? now : initial.LastUpdatedUtc + }; + + var tx = _db.CreateTransaction(); + tx.AddCondition(Condition.KeyNotExists(JobKey(state.JobId))); + _ = tx.HashSetAsync(JobKey(state.JobId), ToHash(state)); + _ = tx.SetAddAsync(StatusKey(state.Status), state.JobId); + _ = tx.SetAddAsync(NameKey(state.Name), state.JobId); + _ = tx.SetAddAsync(AllKey, state.JobId); + return tx.ExecuteAsync(); // result ignored: false => already present; create-if-absent is a no-op + } + + public async Task GetAsync(string jobId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + var entries = await _db.HashGetAllAsync(JobKey(jobId)).ConfigureAwait(false); + return entries.Length == 0 ? null : FromHash(entries); + } + + public async Task> QueryAsync(JobQuery query, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(query); + cancellationToken.ThrowIfCancellationRequested(); + + RedisValue[] ids; + if (query.Status is { } status && !String.IsNullOrEmpty(query.Name)) + ids = await _db.SetCombineAsync(SetOperation.Intersect, StatusKey(status), NameKey(query.Name)).ConfigureAwait(false); + else if (query.Status is { } onlyStatus) + ids = await _db.SetMembersAsync(StatusKey(onlyStatus)).ConfigureAwait(false); + else if (!String.IsNullOrEmpty(query.Name)) + ids = await _db.SetMembersAsync(NameKey(query.Name)).ConfigureAwait(false); + else + ids = await _db.SetMembersAsync(AllKey).ConfigureAwait(false); + + var states = await LoadAsync(ids).ConfigureAwait(false); + return states + // ScheduledForUtc must be filtered after hydration (it isn't indexed), mirroring GetExpiredProcessingAsync. + .Where(s => !query.ExcludeOccurrences || s.ScheduledForUtc is null) + .OrderByDescending(s => s.LastUpdatedUtc) + .Take(Math.Max(1, query.Limit)) + .ToArray(); + } + + public Task TryTransitionAsync(string jobId, JobStatus expectedStatus, JobStatus newStatus, JobStatePatch? patch = null, string? expectedNodeId = null, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + var tx = _db.CreateTransaction(); + tx.AddCondition(Condition.HashEqual(JobKey(jobId), "status", expectedStatus.ToString())); + if (expectedNodeId is not null) + tx.AddCondition(Condition.HashEqual(JobKey(jobId), "nodeId", expectedNodeId)); + + ApplyTransition(tx, jobId, expectedStatus, newStatus, patch); + return tx.ExecuteAsync(); + } + + public async Task TryClaimAsync(string jobId, string nodeId, TimeSpan lease, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + ArgumentException.ThrowIfNullOrEmpty(nodeId); + + var now = _timeProvider.GetUtcNow(); + var current = await _db.HashGetAsync(JobKey(jobId), ["nodeId", "leaseExpiresUtc"]).ConfigureAwait(false); + if (!await _db.KeyExistsAsync(JobKey(jobId)).ConfigureAwait(false)) + return false; + + string? owner = ToStringOrNull(current[0]); + var leaseExpires = ParseTime(current[1]); + bool heldByOther = !String.IsNullOrEmpty(owner) && owner != nodeId && leaseExpires is { } e && e > now; + if (heldByOther) + return false; + + var tx = _db.CreateTransaction(); + if (String.IsNullOrEmpty(owner)) + { + tx.AddCondition(Condition.HashNotExists(JobKey(jobId), "nodeId")); + } + else + { + // Stealing an expired lease: predicate on BOTH the observed owner and the exact lease value, so a + // concurrent renew by that owner (which rewrites leaseExpiresUtc) invalidates the steal and can't + // double-run. Mirrors TryReclaimExpiredAsync; the unguarded version could overwrite a freshly-renewed lease. + tx.AddCondition(Condition.HashEqual(JobKey(jobId), "nodeId", owner)); + if (!current[1].IsNullOrEmpty) + tx.AddCondition(Condition.HashEqual(JobKey(jobId), "leaseExpiresUtc", current[1])); + } + _ = tx.HashSetAsync(JobKey(jobId), + [ + new HashEntry("nodeId", nodeId), + new HashEntry("leaseExpiresUtc", Ticks(now.Add(lease))), + new HashEntry("lastUpdatedUtc", Ticks(now)) + ]); + return await tx.ExecuteAsync().ConfigureAwait(false); + } + + public Task RenewClaimAsync(string jobId, string nodeId, TimeSpan lease, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + var now = _timeProvider.GetUtcNow(); + + var tx = _db.CreateTransaction(); + tx.AddCondition(Condition.HashEqual(JobKey(jobId), "nodeId", nodeId)); + _ = tx.HashSetAsync(JobKey(jobId), + [ + new HashEntry("leaseExpiresUtc", Ticks(now.Add(lease))), + new HashEntry("lastUpdatedUtc", Ticks(now)) + ]); + return tx.ExecuteAsync(); + } + + public Task ReleaseClaimAsync(string jobId, string nodeId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + var tx = _db.CreateTransaction(); + tx.AddCondition(Condition.HashEqual(JobKey(jobId), "nodeId", nodeId)); + _ = tx.HashDeleteAsync(JobKey(jobId), ["nodeId", "leaseExpiresUtc"]); + _ = tx.HashSetAsync(JobKey(jobId), "lastUpdatedUtc", Ticks(_timeProvider.GetUtcNow())); + return tx.ExecuteAsync(); + } + + public async Task> GetExpiredProcessingAsync(DateTimeOffset now, int limit, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + var ids = await _db.SetMembersAsync(StatusKey(JobStatus.Processing)).ConfigureAwait(false); + var states = await LoadAsync(ids).ConfigureAwait(false); + return states + // Exclude CRON occurrences (ScheduledForUtc set): the scheduler owns their recovery. + .Where(s => s.ScheduledForUtc is null && s.LeaseExpiresUtc is { } lease && lease <= now) + .OrderBy(s => s.LeaseExpiresUtc) + .Take(Math.Max(1, limit)) + .ToArray(); + } + + public async Task TryReclaimExpiredAsync(string jobId, DateTimeOffset now, string expectedNodeId, JobStatus newStatus, JobStatePatch? patch = null, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + ArgumentException.ThrowIfNullOrEmpty(expectedNodeId); + + var state = await GetAsync(jobId, cancellationToken).ConfigureAwait(false); + if (state is null || state.Status != JobStatus.Processing || !String.Equals(state.NodeId, expectedNodeId, StringComparison.Ordinal)) + return false; + if (state.LeaseExpiresUtc is not { } lease || lease > now) + return false; + + var tx = _db.CreateTransaction(); + tx.AddCondition(Condition.HashEqual(JobKey(jobId), "status", JobStatus.Processing.ToString())); + tx.AddCondition(Condition.HashEqual(JobKey(jobId), "nodeId", expectedNodeId)); + // Predicate on the exact lease we read; a concurrent renew changes it and invalidates the reclaim. + tx.AddCondition(Condition.HashEqual(JobKey(jobId), "leaseExpiresUtc", Ticks(lease))); + + ApplyTransition(tx, jobId, JobStatus.Processing, newStatus, patch); + return await tx.ExecuteAsync().ConfigureAwait(false); + } + + public Task SetProgressAsync(string jobId, int? percent = null, string? message = null, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + var tx = _db.CreateTransaction(); + tx.AddCondition(Condition.KeyExists(JobKey(jobId))); + if (percent is { } p) + _ = tx.HashSetAsync(JobKey(jobId), "progress", p); + if (message is not null) + _ = tx.HashSetAsync(JobKey(jobId), "progressMessage", message); + _ = tx.HashSetAsync(JobKey(jobId), "lastUpdatedUtc", Ticks(_timeProvider.GetUtcNow())); + return tx.ExecuteAsync(); + } + + public Task IncrementAttemptAsync(string jobId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + var tx = _db.CreateTransaction(); + tx.AddCondition(Condition.KeyExists(JobKey(jobId))); + _ = tx.HashIncrementAsync(JobKey(jobId), "attempt", 1); + _ = tx.HashSetAsync(JobKey(jobId), "lastUpdatedUtc", Ticks(_timeProvider.GetUtcNow())); + return tx.ExecuteAsync(); + } + + public Task RequestCancellationAsync(string jobId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + var tx = _db.CreateTransaction(); + tx.AddCondition(Condition.KeyExists(JobKey(jobId))); + _ = tx.HashSetAsync(JobKey(jobId), + [ + new HashEntry("cancellationRequested", "1"), + new HashEntry("lastUpdatedUtc", Ticks(_timeProvider.GetUtcNow())) + ]); + return tx.ExecuteAsync(); + } + + public async Task IsCancellationRequestedAsync(string jobId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + var value = await _db.HashGetAsync(JobKey(jobId), "cancellationRequested").ConfigureAwait(false); + return value == "1"; + } + + public Task ScheduleDispatchAsync(ScheduledDispatchState dispatch, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(dispatch); + cancellationToken.ThrowIfCancellationRequested(); + + var tx = _db.CreateTransaction(); + tx.AddCondition(Condition.KeyNotExists(DispatchKey(dispatch.DispatchId))); + _ = tx.HashSetAsync(DispatchKey(dispatch.DispatchId), ToHash(dispatch)); + _ = tx.SortedSetAddAsync(DueKey, dispatch.DispatchId, dispatch.DueUtc.UtcTicks); + return tx.ExecuteAsync(); // result ignored: false => already scheduled; no-op + } + + public async Task> ClaimDueDispatchesAsync(DateTimeOffset now, int limit, string nodeId, TimeSpan lease, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + ArgumentException.ThrowIfNullOrEmpty(nodeId); + + var result = await _db.ScriptEvaluateAsync(ClaimDueScript, + [DueKey], + [now.UtcTicks, Math.Max(1, limit), nodeId, Ticks(now.Add(lease)), $"{_prefix}dispatch:"]).ConfigureAwait(false); + + var ids = (RedisValue[]?)result ?? []; + var dispatches = new List(ids.Length); + foreach (var id in ids) + { + var entries = await _db.HashGetAllAsync(DispatchKey(id!)).ConfigureAwait(false); + if (entries.Length > 0) + dispatches.Add(DispatchFromHash(entries)); + } + + return dispatches; + } + + public Task CompleteDispatchAsync(string dispatchId, string nodeId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + var tx = _db.CreateTransaction(); + tx.AddCondition(Condition.HashEqual(DispatchKey(dispatchId), "claimOwner", nodeId)); + _ = tx.KeyDeleteAsync(DispatchKey(dispatchId)); + _ = tx.SortedSetRemoveAsync(DueKey, dispatchId); + return tx.ExecuteAsync(); + } + + public Task ReleaseDispatchAsync(string dispatchId, string nodeId, DateTimeOffset nextDueUtc, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + var tx = _db.CreateTransaction(); + tx.AddCondition(Condition.HashEqual(DispatchKey(dispatchId), "claimOwner", nodeId)); + _ = tx.HashDeleteAsync(DispatchKey(dispatchId), ["claimOwner", "claimExpiresUtc"]); + _ = tx.HashSetAsync(DispatchKey(dispatchId), "dueUtc", Ticks(nextDueUtc)); + _ = tx.SortedSetAddAsync(DueKey, dispatchId, nextDueUtc.UtcTicks); + return tx.ExecuteAsync(); + } + + private void ApplyTransition(ITransaction tx, string jobId, JobStatus fromStatus, JobStatus toStatus, JobStatePatch? patch) + { + var sets = new List + { + new("status", toStatus.ToString()), + new("lastUpdatedUtc", Ticks(patch?.LastUpdatedUtc ?? _timeProvider.GetUtcNow())) + }; + var deletes = new List(); + + if (patch is not null) + { + if (patch.JobType is not null) sets.Add(new("jobType", patch.JobType)); + if (patch.Progress is { } progress) sets.Add(new("progress", progress)); + if (patch.ProgressMessage is not null) sets.Add(new("progressMessage", patch.ProgressMessage)); + if (patch.Error is not null) sets.Add(new("error", patch.Error)); + if (patch.StartedUtc is { } started) sets.Add(new("startedUtc", Ticks(started))); + if (patch.CompletedUtc is { } completed) sets.Add(new("completedUtc", Ticks(completed))); + if (patch.CancellationRequested is { } cancel) sets.Add(new("cancellationRequested", cancel ? "1" : "0")); + + if (patch.ClearNodeId) deletes.Add("nodeId"); + else if (patch.NodeId is not null) sets.Add(new("nodeId", patch.NodeId)); + + if (patch.ClearLeaseExpiresUtc) deletes.Add("leaseExpiresUtc"); + else if (patch.LeaseExpiresUtc is { } leaseExpires) sets.Add(new("leaseExpiresUtc", Ticks(leaseExpires))); + + if (patch.AttemptDelta != 0) + _ = tx.HashIncrementAsync(JobKey(jobId), "attempt", patch.AttemptDelta); + } + + _ = tx.HashSetAsync(JobKey(jobId), sets.ToArray()); + if (deletes.Count > 0) + _ = tx.HashDeleteAsync(JobKey(jobId), deletes.ToArray()); + + if (fromStatus != toStatus) + { + _ = tx.SetRemoveAsync(StatusKey(fromStatus), jobId); + _ = tx.SetAddAsync(StatusKey(toStatus), jobId); + } + } + + private async Task> LoadAsync(RedisValue[] ids) + { + var states = new List(ids.Length); + foreach (var id in ids) + { + var entries = await _db.HashGetAllAsync(JobKey(id!)).ConfigureAwait(false); + if (entries.Length > 0) + states.Add(FromHash(entries)); + } + + return states; + } + + private RedisKey JobKey(string id) => $"{_prefix}job:{id}"; + private RedisKey StatusKey(JobStatus status) => $"{_prefix}status:{status}"; + private RedisKey NameKey(string name) => $"{_prefix}name:{name}"; + private RedisKey DispatchKey(string id) => $"{_prefix}dispatch:{id}"; + private RedisKey AllKey => $"{_prefix}all"; + private RedisKey DueKey => $"{_prefix}dispatches:due"; + + private static string Ticks(DateTimeOffset value) => value.UtcTicks.ToString(CultureInfo.InvariantCulture); + + private static DateTimeOffset? ParseTime(RedisValue value) + { + return value.IsNullOrEmpty || !Int64.TryParse((string?)value, NumberStyles.Integer, CultureInfo.InvariantCulture, out long ticks) + ? null + : new DateTimeOffset(ticks, TimeSpan.Zero); + } + + private static string? ToStringOrNull(RedisValue value) => value.IsNullOrEmpty ? null : (string)value!; + + private static HashEntry[] ToHash(JobState state) + { + var entries = new List + { + new("jobId", state.JobId), + new("name", state.Name), + new("status", state.Status.ToString()), + new("attempt", state.Attempt), + new("cancellationRequested", state.CancellationRequested ? "1" : "0"), + new("createdUtc", Ticks(state.CreatedUtc)), + new("lastUpdatedUtc", Ticks(state.LastUpdatedUtc)) + }; + + if (state.JobType is not null) entries.Add(new("jobType", state.JobType)); + if (state.Payload is { } payload) entries.Add(new("payload", Convert.ToBase64String(payload.Span))); + if (state.PayloadType is not null) entries.Add(new("payloadType", state.PayloadType)); + if (state.Progress is { } progress) entries.Add(new("progress", progress)); + if (state.ProgressMessage is not null) entries.Add(new("progressMessage", state.ProgressMessage)); + if (state.NodeId is not null) entries.Add(new("nodeId", state.NodeId)); + if (state.StartedUtc is { } started) entries.Add(new("startedUtc", Ticks(started))); + if (state.CompletedUtc is { } completed) entries.Add(new("completedUtc", Ticks(completed))); + if (state.LeaseExpiresUtc is { } leaseExpires) entries.Add(new("leaseExpiresUtc", Ticks(leaseExpires))); + if (state.Error is not null) entries.Add(new("error", state.Error)); + if (state.ScheduledForUtc is { } scheduledFor) entries.Add(new("scheduledForUtc", Ticks(scheduledFor))); + + return entries.ToArray(); + } + + private static JobState FromHash(HashEntry[] entries) + { + var map = entries.ToDictionary(e => (string)e.Name!, e => e.Value); + RedisValue Get(string field) => map.TryGetValue(field, out var value) ? value : RedisValue.Null; + + return new JobState + { + JobId = (string)Get("jobId")!, + Name = (string)Get("name")!, + JobType = ToStringOrNull(Get("jobType")), + Payload = Get("payload").IsNullOrEmpty ? null : Convert.FromBase64String((string)Get("payload")!), + PayloadType = ToStringOrNull(Get("payloadType")), + Status = Enum.Parse((string)Get("status")!), + Progress = Get("progress").IsNullOrEmpty ? null : (int)Get("progress"), + ProgressMessage = ToStringOrNull(Get("progressMessage")), + Attempt = Get("attempt").IsNullOrEmpty ? 0 : (int)Get("attempt"), + NodeId = ToStringOrNull(Get("nodeId")), + CreatedUtc = ParseTime(Get("createdUtc")) ?? default, + LastUpdatedUtc = ParseTime(Get("lastUpdatedUtc")) ?? default, + StartedUtc = ParseTime(Get("startedUtc")), + CompletedUtc = ParseTime(Get("completedUtc")), + LeaseExpiresUtc = ParseTime(Get("leaseExpiresUtc")), + Error = ToStringOrNull(Get("error")), + CancellationRequested = Get("cancellationRequested") == "1", + ScheduledForUtc = ParseTime(Get("scheduledForUtc")) + }; + } + + private static HashEntry[] ToHash(ScheduledDispatchState dispatch) + { + var headers = new Dictionary(StringComparer.Ordinal); + foreach (var header in dispatch.Headers) + headers[header.Key] = header.Value; + + var entries = new List + { + new("dispatchId", dispatch.DispatchId), + new("kind", dispatch.Kind.ToString()), + new("body", Convert.ToBase64String(dispatch.Body.Span)), + new("headers", JsonSerializer.Serialize(headers)), + new("options", JsonSerializer.Serialize(dispatch.Options)), + new("dueUtc", Ticks(dispatch.DueUtc)), + new("attempts", dispatch.Attempts) + }; + + // Destination (message dispatches) and JobName (job occurrences) are mutually exclusive; only the populated + // side is written so the read side can distinguish them by field presence. + if (dispatch.Destination is not null) entries.Add(new("destination", JsonSerializer.Serialize(dispatch.Destination))); + if (dispatch.JobName is not null) entries.Add(new("jobName", dispatch.JobName)); + if (dispatch.ClaimOwner is not null) entries.Add(new("claimOwner", dispatch.ClaimOwner)); + if (dispatch.ClaimExpiresUtc is { } claimExpires) entries.Add(new("claimExpiresUtc", Ticks(claimExpires))); + if (dispatch.JobId is not null) entries.Add(new("jobId", dispatch.JobId)); + + return entries.ToArray(); + } + + private static ScheduledDispatchState DispatchFromHash(HashEntry[] entries) + { + var map = entries.ToDictionary(e => (string)e.Name!, e => e.Value); + RedisValue Get(string field) => map.TryGetValue(field, out var value) ? value : RedisValue.Null; + + string headersJson = (string?)Get("headers") ?? "{}"; + var headerMap = JsonSerializer.Deserialize>(headersJson) ?? []; + var options = JsonSerializer.Deserialize((string?)Get("options") ?? "{}") ?? new TransportSendOptions(); + var destination = Get("destination"); + + return new ScheduledDispatchState + { + DispatchId = (string)Get("dispatchId")!, + Kind = Enum.Parse((string)Get("kind")!), + Destination = destination.IsNullOrEmpty ? null : JsonSerializer.Deserialize((string)destination!), + JobName = ToStringOrNull(Get("jobName")), + Body = Get("body").IsNullOrEmpty ? ReadOnlyMemory.Empty : Convert.FromBase64String((string)Get("body")!), + Headers = MessageHeaders.Create(headerMap), + Options = options, + DueUtc = ParseTime(Get("dueUtc")) ?? default, + ClaimOwner = ToStringOrNull(Get("claimOwner")), + ClaimExpiresUtc = ParseTime(Get("claimExpiresUtc")), + Attempts = Get("attempts").IsNullOrEmpty ? 0 : (int)Get("attempts"), + JobId = ToStringOrNull(Get("jobId")) + }; + } +} diff --git a/src/Foundatio.Redis/RedisJobRuntimeStoreOptions.cs b/src/Foundatio.Redis/RedisJobRuntimeStoreOptions.cs new file mode 100644 index 000000000..0675083a2 --- /dev/null +++ b/src/Foundatio.Redis/RedisJobRuntimeStoreOptions.cs @@ -0,0 +1,16 @@ +using System; +using StackExchange.Redis; + +namespace Foundatio.Jobs; + +public class RedisJobRuntimeStoreOptions +{ + /// The Redis connection to use. Required. + public IConnectionMultiplexer ConnectionMultiplexer { get; set; } = null!; + + /// Prefix applied to every key this store creates. Useful to isolate environments/runs on a shared Redis. + public string KeyPrefix { get; set; } = "fnd:jobs:"; + + /// Time source (defaults to ). + public TimeProvider? TimeProvider { get; set; } +} diff --git a/src/Foundatio.TestHarness/Caching/HybridCacheClientTestBase.cs b/src/Foundatio.TestHarness/Caching/HybridCacheClientTestBase.cs index 5963d16c9..c2c77c1cf 100644 --- a/src/Foundatio.TestHarness/Caching/HybridCacheClientTestBase.cs +++ b/src/Foundatio.TestHarness/Caching/HybridCacheClientTestBase.cs @@ -3,7 +3,7 @@ using System.Threading.Tasks; using Foundatio.AsyncEx; using Foundatio.Caching; -using Foundatio.Messaging; +using Foundatio.Messaging.Legacy; using Foundatio.Tests.Extensions; using Microsoft.Extensions.Logging; using Xunit; diff --git a/src/Foundatio.TestHarness/Jobs/HelloWorldJob.cs b/src/Foundatio.TestHarness/Jobs/HelloWorldJob.cs index b79a2a0ff..4ca876c5b 100644 --- a/src/Foundatio.TestHarness/Jobs/HelloWorldJob.cs +++ b/src/Foundatio.TestHarness/Jobs/HelloWorldJob.cs @@ -1,7 +1,7 @@ using System; using System.Threading; using System.Threading.Tasks; -using Foundatio.Jobs; +using Foundatio.Jobs.Legacy; using Microsoft.Extensions.Logging; namespace Foundatio.Tests.Jobs; diff --git a/src/Foundatio.TestHarness/Jobs/JobQueueTestsBase.cs b/src/Foundatio.TestHarness/Jobs/JobQueueTestsBase.cs index 3d1844582..83fd6c5f8 100644 --- a/src/Foundatio.TestHarness/Jobs/JobQueueTestsBase.cs +++ b/src/Foundatio.TestHarness/Jobs/JobQueueTestsBase.cs @@ -6,7 +6,7 @@ using System.Threading.Tasks; using Exceptionless; using Foundatio.Caching; -using Foundatio.Jobs; +using Foundatio.Jobs.Legacy; using Foundatio.Lock; using Foundatio.Queues; using Foundatio.Xunit; diff --git a/src/Foundatio.TestHarness/Jobs/JobRuntimeStoreConformanceTests.cs b/src/Foundatio.TestHarness/Jobs/JobRuntimeStoreConformanceTests.cs new file mode 100644 index 000000000..aac029fa7 --- /dev/null +++ b/src/Foundatio.TestHarness/Jobs/JobRuntimeStoreConformanceTests.cs @@ -0,0 +1,397 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Foundatio.Xunit; +using Microsoft.Extensions.Time.Testing; +using Xunit; + +namespace Foundatio.Tests.Jobs; + +/// +/// Shared conformance suite every implementation must satisfy: state round-trips, +/// optimistic-concurrency transitions, leases/claims, stale recovery (including the renew-during-reclaim race), and +/// scheduled-dispatch claiming. The in-memory reference and any real store (Redis, etc.) run the same assertions so a +/// new backend is validated against the exact behavior the runtime depends on. +/// +/// +/// A drives time so lease-expiry and claim-steal paths are deterministic without real +/// sleeps. returns null when the backing store is unavailable (e.g. Redis not +/// configured), in which case every test skips. +/// +public abstract class JobRuntimeStoreConformanceTests : TestWithLoggingBase +{ + protected JobRuntimeStoreConformanceTests(ITestOutputHelper output) : base(output) { } + + /// Creates a fresh, isolated store bound to , or null when unavailable. + protected abstract IJobRuntimeStore? CreateStore(TimeProvider timeProvider); + + protected static JobState NewJob(TimeProvider time, string id, string name = "conformance-job", JobStatus status = JobStatus.Queued) + { + var now = time.GetUtcNow(); + return new JobState { JobId = id, Name = name, Status = status, CreatedUtc = now, LastUpdatedUtc = now }; + } + + [Fact] + public virtual async Task JobLifecycle_RoundTripsAndTransitionsAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + if (store is null) + { + Assert.Skip("Job runtime store not configured."); + return; + } + + var ct = TestCancellationToken; + var created = time.GetUtcNow(); + + // Create with a rich, fully-populated state and assert every field survives the round-trip. + var job = NewJob(time, "job-1", "emailer") with + { + JobType = "Acme.EmailJob", + Payload = new byte[] { 1, 2, 3, 4 }, + PayloadType = "Acme.EmailJobArgs", + Progress = 10, + ProgressMessage = "starting", + Attempt = 1, + ScheduledForUtc = created.AddMinutes(1) + }; + await store.CreateIfAbsentAsync(job, ct); + + var got = await store.GetAsync("job-1", ct); + Assert.NotNull(got); + Assert.Equal("emailer", got.Name); + Assert.Equal("Acme.EmailJob", got.JobType); + Assert.NotNull(got.Payload); + Assert.Equal(new byte[] { 1, 2, 3, 4 }, got.Payload.Value.ToArray()); + Assert.Equal("Acme.EmailJobArgs", got.PayloadType); + Assert.Equal(JobStatus.Queued, got.Status); + Assert.Equal(10, got.Progress); + Assert.Equal("starting", got.ProgressMessage); + Assert.Equal(1, got.Attempt); + Assert.Equal(created, got.CreatedUtc); + Assert.Equal(created.AddMinutes(1), got.ScheduledForUtc); + + // Create-if-absent is a no-op once the row exists: a second create must not overwrite. + await store.CreateIfAbsentAsync(job with { Name = "overwritten" }, ct); + Assert.Equal("emailer", (await store.GetAsync("job-1", ct))!.Name); + + // A transition from the wrong current status must fail and leave state untouched. + Assert.False(await store.TryTransitionAsync("job-1", JobStatus.Processing, JobStatus.Completed, cancellationToken: ct)); + Assert.Equal(JobStatus.Queued, (await store.GetAsync("job-1", ct))!.Status); + + // Happy-path transition applies the patch atomically (status + node + lease + started + attempt delta). + var lease = time.GetUtcNow().AddMinutes(5); + Assert.True(await store.TryTransitionAsync("job-1", JobStatus.Queued, JobStatus.Processing, + new JobStatePatch { NodeId = "node-a", LeaseExpiresUtc = lease, StartedUtc = created, AttemptDelta = 1 }, cancellationToken: ct)); + got = await store.GetAsync("job-1", ct); + Assert.Equal(JobStatus.Processing, got!.Status); + Assert.Equal("node-a", got.NodeId); + Assert.Equal(lease, got.LeaseExpiresUtc); + Assert.Equal(2, got.Attempt); + Assert.Equal(created, got.StartedUtc); + + // expectedNodeId guards the transition: a stale worker (wrong node) cannot overwrite the owner's state. + Assert.False(await store.TryTransitionAsync("job-1", JobStatus.Processing, JobStatus.Completed, expectedNodeId: "node-b", cancellationToken: ct)); + Assert.Equal(JobStatus.Processing, (await store.GetAsync("job-1", ct))!.Status); + + // Correct owner completes and clears the lease/node. + var completedAt = time.GetUtcNow(); + Assert.True(await store.TryTransitionAsync("job-1", JobStatus.Processing, JobStatus.Completed, + new JobStatePatch { ClearNodeId = true, ClearLeaseExpiresUtc = true, CompletedUtc = completedAt }, expectedNodeId: "node-a", cancellationToken: ct)); + got = await store.GetAsync("job-1", ct); + Assert.Equal(JobStatus.Completed, got!.Status); + Assert.Null(got.NodeId); + Assert.Null(got.LeaseExpiresUtc); + Assert.Equal(completedAt, got.CompletedUtc); + + // Progress, attempt, and cancellation are independent of transitions. + await store.CreateIfAbsentAsync(NewJob(time, "job-2", "worker"), ct); + await store.SetProgressAsync("job-2", 55, "halfway", ct); + await store.IncrementAttemptAsync("job-2", ct); + got = await store.GetAsync("job-2", ct); + Assert.Equal(55, got!.Progress); + Assert.Equal("halfway", got.ProgressMessage); + Assert.Equal(1, got.Attempt); + + Assert.False(await store.IsCancellationRequestedAsync("job-2", ct)); + Assert.True(await store.RequestCancellationAsync("job-2", ct)); + Assert.True(await store.IsCancellationRequestedAsync("job-2", ct)); + + // Operating on a missing job is a benign no-op (returns false / does not throw). + Assert.False(await store.RequestCancellationAsync("missing", ct)); + Assert.Null(await store.GetAsync("missing", ct)); + } + + [Fact] + public virtual async Task Query_FiltersByNameStatusAndLimitAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + if (store is null) + { + Assert.Skip("Job runtime store not configured."); + return; + } + + var ct = TestCancellationToken; + var t = time.GetUtcNow(); + + // Distinct LastUpdatedUtc values make the default newest-first ordering (and limit) deterministic. + await store.CreateIfAbsentAsync(NewJob(time, "a", "alpha", JobStatus.Queued) with { LastUpdatedUtc = t }, ct); + await store.CreateIfAbsentAsync(NewJob(time, "b", "alpha", JobStatus.Processing) with { LastUpdatedUtc = t.AddSeconds(1) }, ct); + await store.CreateIfAbsentAsync(NewJob(time, "c", "beta", JobStatus.Queued) with { LastUpdatedUtc = t.AddSeconds(2) }, ct); + + var byName = await store.QueryAsync(new JobQuery { Name = "alpha" }, ct); + Assert.Equal(["b", "a"], byName.Select(j => j.JobId)); + + var byStatus = await store.QueryAsync(new JobQuery { Status = JobStatus.Queued }, ct); + Assert.Equal(new HashSet { "a", "c" }, byStatus.Select(j => j.JobId).ToHashSet()); + + var byBoth = await store.QueryAsync(new JobQuery { Name = "alpha", Status = JobStatus.Queued }, ct); + Assert.Equal("a", Assert.Single(byBoth).JobId); + + var all = await store.QueryAsync(new JobQuery(), ct); + Assert.Equal(new HashSet { "a", "b", "c" }, all.Select(j => j.JobId).ToHashSet()); + + // Limit is honored against the newest-first ordering, so the most recently updated row wins. + var limited = await store.QueryAsync(new JobQuery { Limit = 1 }, ct); + Assert.Equal("c", Assert.Single(limited).JobId); + + // ExcludeOccurrences filters out CRON occurrences (ScheduledForUtc set) so the generic worker's Queued query + // never claims scheduler-owned jobs. + await store.CreateIfAbsentAsync(NewJob(time, "d", "alpha", JobStatus.Queued) with { LastUpdatedUtc = t.AddSeconds(3), ScheduledForUtc = t }, ct); + var adHocQueued = await store.QueryAsync(new JobQuery { Status = JobStatus.Queued, ExcludeOccurrences = true }, ct); + Assert.Equal(new HashSet { "a", "c" }, adHocQueued.Select(j => j.JobId).ToHashSet()); // "d" excluded (occurrence) + var adHocAlpha = await store.QueryAsync(new JobQuery { Name = "alpha", ExcludeOccurrences = true }, ct); + Assert.Equal(new HashSet { "a", "b" }, adHocAlpha.Select(j => j.JobId).ToHashSet()); // "d" excluded (occurrence) + } + + [Fact] + public virtual async Task Leasing_ClaimRenewReleaseAndStealAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + if (store is null) + { + Assert.Skip("Job runtime store not configured."); + return; + } + + var ct = TestCancellationToken; + await store.CreateIfAbsentAsync(NewJob(time, "job-1"), ct); + + var claimedAt = time.GetUtcNow(); + Assert.True(await store.TryClaimAsync("job-1", "node-a", TimeSpan.FromMinutes(5), ct)); + var got = await store.GetAsync("job-1", ct); + Assert.Equal("node-a", got!.NodeId); + Assert.Equal(claimedAt.AddMinutes(5), got.LeaseExpiresUtc); + + // The current owner can re-claim/renew; a different node cannot while the lease is live. + Assert.True(await store.TryClaimAsync("job-1", "node-a", TimeSpan.FromMinutes(5), ct)); + Assert.False(await store.TryClaimAsync("job-1", "node-b", TimeSpan.FromMinutes(5), ct)); + + // RenewClaim is owner-scoped. + Assert.False(await store.RenewClaimAsync("job-1", "node-b", TimeSpan.FromMinutes(10), ct)); + Assert.True(await store.RenewClaimAsync("job-1", "node-a", TimeSpan.FromMinutes(10), ct)); + Assert.Equal(time.GetUtcNow().AddMinutes(10), (await store.GetAsync("job-1", ct))!.LeaseExpiresUtc); + + // A renewed lease is not stealable: after the lease would have lapsed the owner renews, so a competing steal + // must fail rather than act on a stale expired-lease observation (the steal CAS must see the renew → no double-run). + time.Advance(TimeSpan.FromMinutes(11)); + Assert.True(await store.RenewClaimAsync("job-1", "node-a", TimeSpan.FromMinutes(10), ct)); + Assert.False(await store.TryClaimAsync("job-1", "node-b", TimeSpan.FromMinutes(5), ct)); + Assert.Equal("node-a", (await store.GetAsync("job-1", ct))!.NodeId); + + // Once the renewed lease itself lapses, another node may steal the claim. + time.Advance(TimeSpan.FromMinutes(11)); + Assert.True(await store.TryClaimAsync("job-1", "node-b", TimeSpan.FromMinutes(5), ct)); + Assert.Equal("node-b", (await store.GetAsync("job-1", ct))!.NodeId); + + // Release is owner-scoped and clears the lease. + Assert.False(await store.ReleaseClaimAsync("job-1", "node-a", ct)); + Assert.True(await store.ReleaseClaimAsync("job-1", "node-b", ct)); + got = await store.GetAsync("job-1", ct); + Assert.Null(got!.NodeId); + Assert.Null(got.LeaseExpiresUtc); + } + + [Fact] + public virtual async Task StaleRecovery_ReclaimsExpiredButNotLiveOrCronAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + if (store is null) + { + Assert.Skip("Job runtime store not configured."); + return; + } + + var ct = TestCancellationToken; + var now = time.GetUtcNow(); + + JobState Processing(string id, DateTimeOffset lease, string node = "node-a", DateTimeOffset? scheduledFor = null) => + NewJob(time, id, "worker", JobStatus.Processing) with { NodeId = node, LeaseExpiresUtc = lease, ScheduledForUtc = scheduledFor }; + + await store.CreateIfAbsentAsync(Processing("plain", now.AddMinutes(-1)), ct); + await store.CreateIfAbsentAsync(Processing("cron", now.AddMinutes(-1), scheduledFor: now), ct); + await store.CreateIfAbsentAsync(Processing("live", now.AddMinutes(10)), ct); + + // Only the plain expired job is recoverable: the live lease and the CRON occurrence are excluded. + var expired = await store.GetExpiredProcessingAsync(now, 100, ct); + Assert.Equal("plain", Assert.Single(expired).JobId); + + // Reclaim re-queues it (still owned by node-a, lease still expired). + Assert.True(await store.TryReclaimExpiredAsync("plain", now, "node-a", JobStatus.Queued, + new JobStatePatch { ClearNodeId = true, ClearLeaseExpiresUtc = true, AttemptDelta = 1 }, ct)); + var got = await store.GetAsync("plain", ct); + Assert.Equal(JobStatus.Queued, got!.Status); + Assert.Null(got.NodeId); + Assert.Equal(1, got.Attempt); + + // Renew-during-reclaim race: a job whose owner renewed since the scan must NOT be reclaimed (lease no longer expired). + await store.CreateIfAbsentAsync(Processing("renewed", now.AddMinutes(-1)), ct); + Assert.True(await store.RenewClaimAsync("renewed", "node-a", TimeSpan.FromMinutes(10), ct)); + Assert.False(await store.TryReclaimExpiredAsync("renewed", now, "node-a", JobStatus.Queued, cancellationToken: ct)); + Assert.Equal(JobStatus.Processing, (await store.GetAsync("renewed", ct))!.Status); + + // Owner mismatch since the scan also blocks the reclaim. + await store.CreateIfAbsentAsync(Processing("reowned", now.AddMinutes(-1), node: "node-b"), ct); + Assert.False(await store.TryReclaimExpiredAsync("reowned", now, "node-a", JobStatus.Queued, cancellationToken: ct)); + Assert.Equal("node-b", (await store.GetAsync("reowned", ct))!.NodeId); + } + + [Fact] + public virtual async Task ScheduledDispatches_ClaimCompleteAndRescheduleAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + if (store is null) + { + Assert.Skip("Job runtime store not configured."); + return; + } + + var ct = TestCancellationToken; + var t = time.GetUtcNow(); + + var headers = MessageHeaders.Create(new Dictionary { ["message.type"] = "order.created", ["tenant"] = "acme" }); + var options = new TransportSendOptions { Priority = MessagePriority.High }; + byte[] body = [0x01, 0x02, 0xFF, 0x00, 0x10]; + + var due = new ScheduledDispatchState + { + DispatchId = "d1", + Kind = ScheduledDispatchKind.JobOccurrence, + JobName = "jobs", + Body = body, + Headers = headers, + Options = options, + DueUtc = t.AddMinutes(-1), + JobId = "job-x" + }; + var future = new ScheduledDispatchState + { + DispatchId = "d2", + Kind = ScheduledDispatchKind.QueueMessage, + Destination = DestinationAddress.ForQueue("later"), + Body = body, + DueUtc = t.AddHours(1) + }; + await store.ScheduleDispatchAsync(due, ct); + await store.ScheduleDispatchAsync(future, ct); + // Re-scheduling the same id is a no-op (must not overwrite the dispatch). + await store.ScheduleDispatchAsync(due with { JobName = "overwritten" }, ct); + + // Only the due dispatch is claimed; the full payload round-trips and the attempt counter increments. + var claimed = await store.ClaimDueDispatchesAsync(t, 100, "node-a", TimeSpan.FromMinutes(5), ct); + var d = Assert.Single(claimed); + Assert.Equal("d1", d.DispatchId); + Assert.Equal(ScheduledDispatchKind.JobOccurrence, d.Kind); + Assert.Equal("jobs", d.JobName); + Assert.Equal(body, d.Body.ToArray()); + Assert.Equal("acme", d.Headers["tenant"]); + Assert.Equal("order.created", d.Headers["message.type"]); + Assert.Equal(MessagePriority.High, d.Options.Priority); + Assert.Equal("node-a", d.ClaimOwner); + Assert.Equal(1, d.Attempts); + Assert.Equal("job-x", d.JobId); + + // A competing claim sees nothing while the lease is live (and d2 is not yet due). + Assert.Empty(await store.ClaimDueDispatchesAsync(t, 100, "node-b", TimeSpan.FromMinutes(5), ct)); + + // A complete from the wrong owner is ignored: after the lease lapses the dispatch is re-claimable, attempt 2. + await store.CompleteDispatchAsync("d1", "node-b", ct); + var reclaimed = await store.ClaimDueDispatchesAsync(t.AddMinutes(6), 100, "node-a", TimeSpan.FromMinutes(5), ct); + Assert.Equal(2, Assert.Single(reclaimed).Attempts); + + // The owning node completes it for good. + await store.CompleteDispatchAsync("d1", "node-a", ct); + Assert.Empty(await store.ClaimDueDispatchesAsync(t.AddMinutes(12), 100, "node-a", TimeSpan.FromMinutes(5), ct)); + + // Release reschedules a claimed dispatch to its next due time and clears ownership (recurring-occurrence path). + var recurring = new ScheduledDispatchState + { + DispatchId = "d3", + Kind = ScheduledDispatchKind.JobOccurrence, + JobName = "cron", + Body = body, + DueUtc = t.AddMinutes(20) + }; + await store.ScheduleDispatchAsync(recurring, ct); + Assert.Equal("d3", Assert.Single(await store.ClaimDueDispatchesAsync(t.AddMinutes(21), 100, "node-a", TimeSpan.FromMinutes(5), ct)).DispatchId); + + await store.ReleaseDispatchAsync("d3", "node-b", t.AddMinutes(50), ct); // wrong owner: ignored + await store.ReleaseDispatchAsync("d3", "node-a", t.AddMinutes(50), ct); + Assert.Empty(await store.ClaimDueDispatchesAsync(t.AddMinutes(40), 100, "node-c", TimeSpan.FromMinutes(5), ct)); + var rescheduled = Assert.Single(await store.ClaimDueDispatchesAsync(t.AddMinutes(51), 100, "node-c", TimeSpan.FromMinutes(5), ct)); + Assert.Equal("d3", rescheduled.DispatchId); + Assert.Equal("node-c", rescheduled.ClaimOwner); + } + + [Fact] + public virtual async Task Concurrency_OptimisticControlElectsSingleWinnerAsync() + { + var time = new FakeTimeProvider(); + var store = CreateStore(time); + if (store is null) + { + Assert.Skip("Job runtime store not configured."); + return; + } + + var ct = TestCancellationToken; + const int contenders = 25; + + // Many nodes race to claim the same unclaimed job: exactly one may win, and the store must agree on the owner. + await store.CreateIfAbsentAsync(NewJob(time, "claim-race"), ct); + var claims = await Task.WhenAll(Enumerable.Range(0, contenders) + .Select(i => Task.Run(() => store.TryClaimAsync("claim-race", $"node-{i}", TimeSpan.FromMinutes(5), ct), ct))); + Assert.Equal(1, claims.Count(won => won)); + var ownedBy = (await store.GetAsync("claim-race", ct))!.NodeId; + Assert.StartsWith("node-", ownedBy); + + // Many nodes race the same Queued -> Processing transition: optimistic concurrency must admit exactly one. + await store.CreateIfAbsentAsync(NewJob(time, "transition-race"), ct); + var transitions = await Task.WhenAll(Enumerable.Range(0, contenders) + .Select(i => Task.Run(() => store.TryTransitionAsync("transition-race", JobStatus.Queued, JobStatus.Processing, + new JobStatePatch { NodeId = $"node-{i}" }, cancellationToken: ct), ct))); + Assert.Equal(1, transitions.Count(won => won)); + Assert.Equal(JobStatus.Processing, (await store.GetAsync("transition-race", ct))!.Status); + + // A single due dispatch contested by many claimers must be handed to exactly one. + await store.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = "dispatch-race", + Destination = DestinationAddress.ForQueue("q"), + Body = new byte[] { 1 }, + DueUtc = time.GetUtcNow().AddMinutes(-1) + }, ct); + var dispatchClaims = await Task.WhenAll(Enumerable.Range(0, contenders) + .Select(i => Task.Run(() => store.ClaimDueDispatchesAsync(time.GetUtcNow(), 100, $"node-{i}", TimeSpan.FromMinutes(5), ct), ct))); + Assert.Equal(1, dispatchClaims.Sum(claimed => claimed.Count(d => d.DispatchId == "dispatch-race"))); + } +} diff --git a/src/Foundatio.TestHarness/Jobs/SampleQueueJob.cs b/src/Foundatio.TestHarness/Jobs/SampleQueueJob.cs index c5dfde3de..d03aac214 100644 --- a/src/Foundatio.TestHarness/Jobs/SampleQueueJob.cs +++ b/src/Foundatio.TestHarness/Jobs/SampleQueueJob.cs @@ -2,7 +2,7 @@ using System.Threading; using System.Threading.Tasks; using Exceptionless; -using Foundatio.Jobs; +using Foundatio.Jobs.Legacy; using Foundatio.Lock; using Foundatio.Queues; using Foundatio.Resilience; diff --git a/src/Foundatio.TestHarness/Jobs/ThrottledJob.cs b/src/Foundatio.TestHarness/Jobs/ThrottledJob.cs index bdbfb56af..5d5e84e1d 100644 --- a/src/Foundatio.TestHarness/Jobs/ThrottledJob.cs +++ b/src/Foundatio.TestHarness/Jobs/ThrottledJob.cs @@ -2,7 +2,7 @@ using System.Threading; using System.Threading.Tasks; using Foundatio.Caching; -using Foundatio.Jobs; +using Foundatio.Jobs.Legacy; using Foundatio.Lock; using Microsoft.Extensions.Logging; diff --git a/src/Foundatio.TestHarness/Jobs/WithDependencyJob.cs b/src/Foundatio.TestHarness/Jobs/WithDependencyJob.cs index 054b123a0..b205b123c 100644 --- a/src/Foundatio.TestHarness/Jobs/WithDependencyJob.cs +++ b/src/Foundatio.TestHarness/Jobs/WithDependencyJob.cs @@ -1,5 +1,5 @@ using System.Threading.Tasks; -using Foundatio.Jobs; +using Foundatio.Jobs.Legacy; using Microsoft.Extensions.Logging; namespace Foundatio.Tests.Jobs; diff --git a/src/Foundatio.TestHarness/Jobs/WithLockingJob.cs b/src/Foundatio.TestHarness/Jobs/WithLockingJob.cs index 9bbfd120f..3c31e39ab 100644 --- a/src/Foundatio.TestHarness/Jobs/WithLockingJob.cs +++ b/src/Foundatio.TestHarness/Jobs/WithLockingJob.cs @@ -2,9 +2,9 @@ using System.Threading; using System.Threading.Tasks; using Foundatio.Caching; -using Foundatio.Jobs; +using Foundatio.Jobs.Legacy; using Foundatio.Lock; -using Foundatio.Messaging; +using Foundatio.Messaging.Legacy; using Microsoft.Extensions.Logging; using Xunit; diff --git a/src/Foundatio.TestHarness/Messaging/MessageBusTestBase.cs b/src/Foundatio.TestHarness/Messaging/MessageBusTestBase.cs index 523fa5bf3..dc492a1f5 100644 --- a/src/Foundatio.TestHarness/Messaging/MessageBusTestBase.cs +++ b/src/Foundatio.TestHarness/Messaging/MessageBusTestBase.cs @@ -6,7 +6,7 @@ using System.Threading.Tasks; using Exceptionless; using Foundatio.AsyncEx; -using Foundatio.Messaging; +using Foundatio.Messaging.Legacy; using Foundatio.Tests.Extensions; using Foundatio.Tests.Serializer; using Foundatio.Tests.Utility; diff --git a/src/Foundatio.TestHarness/Messaging/MessageTransportConformanceTests.cs b/src/Foundatio.TestHarness/Messaging/MessageTransportConformanceTests.cs new file mode 100644 index 000000000..f2f3be2b3 --- /dev/null +++ b/src/Foundatio.TestHarness/Messaging/MessageTransportConformanceTests.cs @@ -0,0 +1,718 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Foundatio.Xunit; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public abstract class MessageTransportConformanceTests : TestWithLoggingBase +{ + protected MessageTransportConformanceTests(ITestOutputHelper output) : base(output) { } + + protected virtual IMessageTransport? CreateTransport() + { + return null; + } + + protected virtual ValueTask CleanupTransportAsync(IMessageTransport transport) + { + return transport.DisposeAsync(); + } + + [Fact] + public virtual async Task CanSendAndReceiveBatchAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull) + { + Assert.Skip("Transport does not support pull receive (ISupportsPull)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("orders"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + + var result = await transport.SendAsync(queue, [ + CreateMessage("one", ("tenant", "acme")), + CreateMessage("two", ("tenant", "acme")) + ], new TransportSendOptions(), TestCancellationToken); + + // Send is throw-on-failure, so reaching here means both messages were accepted; assert the accepted ids. + Assert.Equal(2, result.Items.Count); + + var entries = await pull.ReceiveAsync(queue, new ReceiveRequest + { + MaxMessages = 2, + MaxWaitTime = TimeSpan.FromSeconds(1) + }, TestCancellationToken); + + Assert.Equal(2, entries.Count); + var bodies = entries.Select(ReadBody).ToList(); + Assert.Contains("one", bodies); + Assert.Contains("two", bodies); + + // Only assert positional FIFO order when the transport actually guarantees ordering; a best-effort + // (OrderingGuarantee.None) transport may legitimately deliver out of order. + if (GetCapabilities(transport, DestinationRole.Queue).Ordering != OrderingGuarantee.None) + { + Assert.Equal("one", ReadBody(entries[0])); + Assert.Equal("two", ReadBody(entries[1])); + } + + Assert.All(entries, e => Assert.Equal("acme", e.Headers["tenant"])); + Assert.Equal(1, entries[0].DeliveryCount); + + await transport.CompleteAsync(entries[0], TestCancellationToken); + await transport.CompleteAsync(entries[1], TestCancellationToken); + + if (transport is ISupportsStats stats) + { + // Assert only the point-in-time gauges every broker can report, and tolerate eventual consistency + // (e.g. SQS ApproximateNumberOf* lag). Lifetime counters such as Completed are not universally + // available across transports, so they are not part of the shared contract. + await AssertQueueDrainedAsync(stats, queue, TestCancellationToken); + } + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task AbandonAsync_RedeliversWithIncrementedDeliveryCountAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull) + { + Assert.Skip("Transport does not support pull receive (ISupportsPull)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("retry"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("retry-me")], new TransportSendOptions(), TestCancellationToken); + + var first = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(1) }, TestCancellationToken)); + Assert.Equal(1, first.DeliveryCount); + + await transport.AbandonAsync(first, TestCancellationToken); + + var second = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(1) }, TestCancellationToken)); + Assert.Equal(first.Id, second.Id); + Assert.Equal(2, second.DeliveryCount); + Assert.Equal("retry-me", ReadBody(second)); + + await transport.CompleteAsync(second, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task CompleteAsync_WithExpiredReceipt_ThrowsReceiptExpiredExceptionAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull) + { + Assert.Skip("Transport does not support pull receive (ISupportsPull)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("receipts"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("done")], new TransportSendOptions(), TestCancellationToken); + + var entry = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(1) }, TestCancellationToken)); + await transport.CompleteAsync(entry, TestCancellationToken); + + await Assert.ThrowsAsync(async () => + await transport.CompleteAsync(entry, TestCancellationToken)); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task SubscribeAsync_DeliversPushMessagesAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPush push) + { + Assert.Skip("Transport does not support push delivery (ISupportsPush)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("push"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + + var received = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await using var subscription = await push.SubscribeAsync(queue, async (entry, ct) => + { + await transport.CompleteAsync(entry, ct); + received.TrySetResult(entry); + }, new PushOptions(), TestCancellationToken); + + await transport.SendAsync(queue, [CreateMessage("pushed")], new TransportSendOptions(), TestCancellationToken); + + var completed = await Task.WhenAny(received.Task, Task.Delay(TimeSpan.FromSeconds(3), TestCancellationToken)); + Assert.Equal(received.Task, completed); + Assert.Equal("pushed", ReadBody(await received.Task)); + Assert.Equal(queue, subscription.Source); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task SendAsync_ToTopic_FansOutToSubscriptionsAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull || transport is not ISupportsProvisioning) + { + Assert.Skip("Transport does not support pull receive and provisioning (ISupportsPull + ISupportsProvisioning)."); + return; + } + + try + { + var topic = DestinationAddress.ForTopic("orders-topic"); + var subscriptionA = DestinationAddress.ForSubscription("orders-topic", "orders-subscription-a"); + var subscriptionB = DestinationAddress.ForSubscription("orders-topic", "orders-subscription-b"); + await EnsureAsync(transport, + new DestinationDeclaration { Address = topic }, + new DestinationDeclaration { Address = subscriptionA }, + new DestinationDeclaration { Address = subscriptionB }); + + // The address states the destination role; publishing to a topic must use a topic-role address. + await transport.SendAsync(topic, [CreateMessage("fanout")], new TransportSendOptions(), TestCancellationToken); + + var first = Assert.Single(await pull.ReceiveAsync(subscriptionA, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, TestCancellationToken)); + var second = Assert.Single(await pull.ReceiveAsync(subscriptionB, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(1) }, TestCancellationToken)); + + Assert.Equal("fanout", ReadBody(first)); + Assert.Equal("fanout", ReadBody(second)); + + await transport.CompleteAsync(first, TestCancellationToken); + await transport.CompleteAsync(second, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task ProvisioningLifecycle_EnsureExistsDeleteAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsProvisioning provisioning) + { + Assert.Skip("Transport does not support provisioning (ISupportsProvisioning)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("prov-queue"); + var topic = DestinationAddress.ForTopic("prov-topic"); + var subscription = DestinationAddress.ForSubscription("prov-topic", "prov-sub"); + + Assert.False(await provisioning.ExistsAsync(queue, TestCancellationToken)); + Assert.False(await provisioning.ExistsAsync(topic, TestCancellationToken)); + Assert.False(await provisioning.ExistsAsync(subscription, TestCancellationToken)); + + DestinationDeclaration[] declarations = [ + new DestinationDeclaration { Address = queue }, + new DestinationDeclaration { Address = topic }, + new DestinationDeclaration { Address = subscription } + ]; + await provisioning.EnsureAsync(declarations, TestCancellationToken); + + Assert.True(await provisioning.ExistsAsync(queue, TestCancellationToken)); + Assert.True(await provisioning.ExistsAsync(topic, TestCancellationToken)); + Assert.True(await provisioning.ExistsAsync(subscription, TestCancellationToken)); + + // Ensure is idempotent: re-declaring destinations that already exist must not throw. + await provisioning.EnsureAsync(declarations, TestCancellationToken); + + await provisioning.DeleteAsync(subscription, TestCancellationToken); + await provisioning.DeleteAsync(topic, TestCancellationToken); + await provisioning.DeleteAsync(queue, TestCancellationToken); + + Assert.False(await provisioning.ExistsAsync(queue, TestCancellationToken)); + Assert.False(await provisioning.ExistsAsync(topic, TestCancellationToken)); + Assert.False(await provisioning.ExistsAsync(subscription, TestCancellationToken)); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task SendAsync_ToTopic_WithDeliverAt_WithoutNativeDelay_ThrowsAsync() + { + var transport = CreateTransport(); + if (transport is null) + { + Assert.Skip("No transport configured."); + return; + } + + if (transport is ITransportInfo { SupportedRoles: { } roles } && !roles.Contains(DestinationRole.Topic)) + { + Assert.Skip("Transport does not support topic destinations."); + return; + } + + if (GetCapabilities(transport, DestinationRole.Topic).DelayedDelivery) + { + Assert.Skip("Transport honors delayed delivery natively for topics; nothing to refuse."); + return; + } + + try + { + // A transport that cannot honor DeliverAt for a role must refuse it, never publish immediately and + // silently drop the delay — the core only routes a delayed send here when the role advertises the + // capability, so acceptance would mean a lost delay (the AWS SNS delayed-publish bug shape). + await Assert.ThrowsAsync(() => transport.SendAsync(DestinationAddress.ForTopic("delayed-topic"), + [CreateMessage("later")], + new TransportSendOptions { DeliverAt = DateTimeOffset.UtcNow.AddMinutes(5) }, + TestCancellationToken)); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task ReceiveAsync_RespectsPriorityAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull || !GetCapabilities(transport, DestinationRole.Queue).Priority) + { + Assert.Skip("Transport does not support pull receive with queue priority (ISupportsPull + Priority capability)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("priority"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("low")], new TransportSendOptions { Priority = MessagePriority.Low }, TestCancellationToken); + await transport.SendAsync(queue, [CreateMessage("high")], new TransportSendOptions { Priority = MessagePriority.High }, TestCancellationToken); + await transport.SendAsync(queue, [CreateMessage("normal")], new TransportSendOptions { Priority = MessagePriority.Normal }, TestCancellationToken); + + var entries = await pull.ReceiveAsync(queue, new ReceiveRequest + { + MaxMessages = 3, + MaxWaitTime = TimeSpan.FromSeconds(1) + }, TestCancellationToken); + + Assert.Equal(3, entries.Count); + Assert.Equal("high", ReadBody(entries[0])); + Assert.Equal("normal", ReadBody(entries[1])); + Assert.Equal("low", ReadBody(entries[2])); + + foreach (var entry in entries) + await transport.CompleteAsync(entry, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task SendAsync_WithDeliverAt_DelaysVisibilityAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull || !GetCapabilities(transport, DestinationRole.Queue).DelayedDelivery) + { + Assert.Skip("Transport does not support pull receive with native queue delayed delivery (ISupportsPull + DelayedDelivery capability)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("delayed"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("later")], new TransportSendOptions + { + DeliverAt = DateTimeOffset.UtcNow.AddMilliseconds(250) + }, TestCancellationToken); + + var immediate = await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromMilliseconds(50) }, TestCancellationToken); + Assert.Empty(immediate); + + var delayed = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, TestCancellationToken)); + Assert.Equal("later", ReadBody(delayed)); + await transport.CompleteAsync(delayed, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task DeadLetterAsync_MovesEntryToDeadletterStatsAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull || transport is not ISupportsDeadLetter || transport is not ISupportsStats stats) + { + Assert.Skip("Transport does not support pull receive with dead-letter and stats (ISupportsPull + ISupportsDeadLetter + ISupportsStats)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("deadletter"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("poison")], new TransportSendOptions(), TestCancellationToken); + + var entry = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(1) }, TestCancellationToken)); + await ((ISupportsDeadLetter)transport).DeadLetterAsync(entry, "bad-payload", TestCancellationToken); + + MessageDestinationStats queueStats = await stats.GetStatsAsync(queue, TestCancellationToken); + Assert.Equal(0, queueStats.Working); + Assert.Equal(1, queueStats.Deadletter); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task ReceiveAsync_WithExpiredMessage_DeadlettersAndSkipsAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull || !GetCapabilities(transport, DestinationRole.Queue).Expiration || transport is not ISupportsStats stats) + { + Assert.Skip("Transport does not support pull receive with expiration and stats (ISupportsPull + Expiration capability + ISupportsStats)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("expiration"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + var expired = new TransportMessage + { + Body = Encoding.UTF8.GetBytes("expired"), + Headers = MessageHeaders.Create([ + new KeyValuePair(KnownHeaders.Expiration, DateTimeOffset.UtcNow.AddMinutes(-1).ToString("O")) + ]) + }; + + await transport.SendAsync(queue, [expired], new TransportSendOptions(), TestCancellationToken); + + var entries = await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromMilliseconds(50) }, TestCancellationToken); + Assert.Empty(entries); + + MessageDestinationStats queueStats = await stats.GetStatsAsync(queue, TestCancellationToken); + Assert.Equal(0, queueStats.Queued); + Assert.Equal(1, queueStats.Deadletter); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + [Fact] + public virtual async Task ReceiveAsync_AfterVisibilityTimeout_RedeliversAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsVisibilityTimeout visibility) + { + Assert.Skip("Transport does not support visibility timeout (ISupportsVisibilityTimeout)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("visibility"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("lease")], new TransportSendOptions(), TestCancellationToken); + + // Whole-second visibility window: real brokers (e.g. SQS) only support second-resolution visibility timeouts. + var visibilityWindow = TimeSpan.FromSeconds(2); + var first = Assert.Single(await visibility.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, visibilityWindow, TestCancellationToken)); + Assert.Equal(1, first.DeliveryCount); + + // Still within the visibility window: a competing receive must not see the in-flight message. + var hidden = await visibility.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromMilliseconds(100) }, visibilityWindow, TestCancellationToken); + Assert.Empty(hidden); + + // After the visibility window lapses without settlement the message must be redelivered (at-least-once). A + // long poll observes the lapse — a transport wakes a blocked receive when a visibility window expires — so + // this is robust to coarse/variable redelivery latency without a fixed sleep. + var second = Assert.Single(await visibility.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = visibilityWindow + TimeSpan.FromSeconds(5) }, visibilityWindow, TestCancellationToken)); + Assert.Equal(first.Id, second.Id); + Assert.Equal(2, second.DeliveryCount); + + await transport.CompleteAsync(second, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task AbandonAsync_WithRedeliveryDelay_RedeliversAfterDelayAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull || transport is not ISupportsRedeliveryDelay redelivery) + { + Assert.Skip("Transport does not support pull receive with redelivery delay (ISupportsPull + ISupportsRedeliveryDelay)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("redelivery-delay"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("delay-me")], new TransportSendOptions(), TestCancellationToken); + + var first = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, TestCancellationToken)); + Assert.Equal(1, first.DeliveryCount); + + // Whole-second redelivery delay: SQS serves this via ChangeMessageVisibility, which is second-resolution. + var redeliveryDelay = TimeSpan.FromSeconds(2); + await redelivery.AbandonAsync(first, redeliveryDelay, TestCancellationToken); + + // Within the delay window the message must not be visible again. + var early = await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromMilliseconds(100) }, TestCancellationToken); + Assert.Empty(early); + + // After the delay lapses it is redelivered with an incremented delivery count. Long poll for robustness. + var second = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = redeliveryDelay + TimeSpan.FromSeconds(5) }, TestCancellationToken)); + Assert.Equal(first.Id, second.Id); + Assert.Equal(2, second.DeliveryCount); + Assert.Equal("delay-me", ReadBody(second)); + + await transport.CompleteAsync(second, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task RenewLockAsync_ExtendsVisibilityWindowAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsVisibilityTimeout visibility || transport is not ISupportsLockRenewal lockRenewal) + { + Assert.Skip("Transport does not support visibility timeout with lock renewal (ISupportsVisibilityTimeout + ISupportsLockRenewal)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("lock-renewal"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("hold")], new TransportSendOptions(), TestCancellationToken); + + // Whole-second windows so the test maps onto second-resolution brokers (e.g. SQS). + var originalWindow = TimeSpan.FromSeconds(2); + var renewedWindow = TimeSpan.FromSeconds(8); + var first = Assert.Single(await visibility.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, originalWindow, TestCancellationToken)); + Assert.Equal(1, first.DeliveryCount); + + // Renew before the original window lapses, extending it well past the original expiry. + await Task.Delay(TimeSpan.FromSeconds(1), TestCancellationToken); + await lockRenewal.RenewLockAsync(first, renewedWindow, TestCancellationToken); + + // Past the original window but inside the renewed window: the message must still be held, so a competing + // receive sees nothing rather than a premature redelivery. + await Task.Delay(originalWindow, TestCancellationToken); + var held = await visibility.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromMilliseconds(100) }, originalWindow, TestCancellationToken); + Assert.Empty(held); + + await transport.CompleteAsync(first, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task CompetingConsumers_DoNotReceiveTheSameInFlightMessageAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull) + { + Assert.Skip("Transport does not support pull receive (ISupportsPull)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("competing"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("once")], new TransportSendOptions(), TestCancellationToken); + + var first = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(1) }, TestCancellationToken)); + + // A competing consumer must not receive the same message while it is in flight. + var second = await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromMilliseconds(100) }, TestCancellationToken); + Assert.Empty(second); + + await transport.CompleteAsync(first, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task ReceiveDeadLetteredAsync_ReturnsPoisonPayloadAndReasonAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull || transport is not ISupportsDeadLetter deadLetter) + { + Assert.Skip("Transport does not support pull receive and dead-letter (ISupportsPull + ISupportsDeadLetter)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("dlq-read"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + await transport.SendAsync(queue, [CreateMessage("poison", ("tenant", "acme"))], new TransportSendOptions(), TestCancellationToken); + + var entry = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(1) }, TestCancellationToken)); + await deadLetter.DeadLetterAsync(entry, "bad-payload", TestCancellationToken); + + // The raw (un-deserialized) payload and the dead-letter reason must be inspectable. + var deadLettered = Assert.Single(await deadLetter.ReceiveDeadLetteredAsync(queue, new ReceiveRequest { MaxMessages = 10 }, TestCancellationToken)); + Assert.Equal("poison", ReadBody(deadLettered)); + Assert.Equal("acme", deadLettered.Headers["tenant"]); + Assert.Equal("bad-payload", deadLettered.Headers[KnownHeaders.DeadLetterReason]); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + [Fact] + public virtual async Task SendAsync_PreservesBinaryBodyAndCaseInsensitiveHeadersAsync() + { + var transport = CreateTransport(); + if (transport is not ISupportsPull pull) + { + Assert.Skip("Transport does not support pull receive (ISupportsPull)."); + return; + } + + try + { + var queue = DestinationAddress.ForQueue("binary"); + await EnsureAsync(transport, new DestinationDeclaration { Address = queue }); + + // Arbitrary, non-UTF-8 bytes with no content type must round-trip exactly (catches body-encoding bugs — a + // provider must not assume text), and header keys must round-trip case-insensitively across the wire. + byte[] payload = [0x00, 0x01, 0xFF, 0xFE, 0x10, 0x80, 0x7F]; + await transport.SendAsync(queue, [new TransportMessage + { + Body = payload, + Headers = MessageHeaders.Create([ + new KeyValuePair("tenant", "acme"), + new KeyValuePair("Mixed.Case", "x") + ]) + }], new TransportSendOptions(), TestCancellationToken); + + var entry = Assert.Single(await pull.ReceiveAsync(queue, new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, TestCancellationToken)); + Assert.Equal(payload, entry.Body.ToArray()); + Assert.Equal("acme", entry.Headers["tenant"]); + Assert.Equal("x", entry.Headers["MIXED.CASE"]); + + await transport.CompleteAsync(entry, TestCancellationToken); + } + finally + { + await CleanupTransportIfNotNullAsync(transport); + } + } + + private async ValueTask CleanupTransportIfNotNullAsync(IMessageTransport? transport) + { + if (transport is not null) + await CleanupTransportAsync(transport); + } + + // Polls until the destination reports no queued or in-flight messages (the point-in-time gauges every broker can + // report), tolerating transports whose stats are only eventually consistent (e.g. SQS ApproximateNumberOf*). + private async Task AssertQueueDrainedAsync(ISupportsStats stats, DestinationAddress destination, CancellationToken cancellationToken) + { + var current = await stats.GetStatsAsync(destination, cancellationToken); + for (int attempt = 0; attempt < 50 && (current.Queued != 0 || current.Working != 0); attempt++) + { + await Task.Delay(100, cancellationToken); + current = await stats.GetStatsAsync(destination, cancellationToken); + } + + Assert.Equal(0, current.Queued); + Assert.Equal(0, current.Working); + } + + private static TransportCapabilities GetCapabilities(IMessageTransport transport, DestinationRole role) + { + return transport is ITransportInfo info ? info.GetCapabilities(role) : TransportCapabilities.None; + } + + private static async Task EnsureAsync(IMessageTransport transport, params DestinationDeclaration[] declarations) + { + if (transport is ISupportsProvisioning provisioning) + await provisioning.EnsureAsync(declarations, CancellationToken.None); + } + + private static TransportMessage CreateMessage(string body, params (string Key, string Value)[] headers) + { + return new TransportMessage + { + Body = Encoding.UTF8.GetBytes(body), + Headers = MessageHeaders.Create(ToKeyValuePairs(headers)) + }; + } + + private static IEnumerable> ToKeyValuePairs((string Key, string Value)[] headers) + { + foreach (var header in headers) + yield return new KeyValuePair(header.Key, header.Value); + } + + private static string ReadBody(TransportEntry entry) + { + return Encoding.UTF8.GetString(entry.Body.Span); + } +} diff --git a/src/Foundatio.TestHarness/Queue/QueueTestBase.cs b/src/Foundatio.TestHarness/Queue/QueueTestBase.cs index 9c8ed42df..01d2f63ca 100644 --- a/src/Foundatio.TestHarness/Queue/QueueTestBase.cs +++ b/src/Foundatio.TestHarness/Queue/QueueTestBase.cs @@ -7,9 +7,9 @@ using Exceptionless; using Foundatio.AsyncEx; using Foundatio.Caching; -using Foundatio.Jobs; +using Foundatio.Jobs.Legacy; using Foundatio.Lock; -using Foundatio.Messaging; +using Foundatio.Messaging.Legacy; using Foundatio.Queues; using Foundatio.Serializer; using Foundatio.Tests.Extensions; diff --git a/src/Foundatio.Testing/Foundatio.Testing.csproj b/src/Foundatio.Testing/Foundatio.Testing.csproj new file mode 100644 index 000000000..1989146f8 --- /dev/null +++ b/src/Foundatio.Testing/Foundatio.Testing.csproj @@ -0,0 +1,8 @@ + + + Test harness for Foundatio messaging: run the real message bus over a recording in-memory transport, await quiescence, and assert on the messages that were sent, published, handled, retried, or dead-lettered. + + + + + diff --git a/src/Foundatio.Testing/MessagingTestHarness.cs b/src/Foundatio.Testing/MessagingTestHarness.cs new file mode 100644 index 000000000..9c093f3c9 --- /dev/null +++ b/src/Foundatio.Testing/MessagingTestHarness.cs @@ -0,0 +1,165 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Text; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Serializer; + +namespace Foundatio.Messaging.Testing; + +/// +/// One recorded message movement: a send/publish accepted by the transport, or a settlement (handled, abandoned for +/// retry, dead-lettered) of a delivered message. +/// +public sealed record RecordedMessage +{ + public required string Destination { get; init; } + + /// + /// The role the message was sent to (Queue for sends, Topic for publishes). Settlement recordings always report + /// Queue: every delivery settles on a queue-shaped channel, including topic deliveries via their subscriptions. + /// + public required DestinationRole Role { get; init; } + + public string? MessageType { get; init; } + public required ReadOnlyMemory Body { get; init; } + public MessageHeaders Headers { get; init; } = MessageHeaders.Empty; + + /// The dead-letter reason, for dead-lettered recordings. + public string? Reason { get; init; } + + /// The delivery count at settlement, for settlement recordings. + public int Attempts { get; init; } +} + +/// +/// Deterministic messaging tests without sleeps: the harness runs the real bus over a recording in-memory transport, +/// so tests act (send/publish), until every queue and in-flight handler drains, then +/// assert on what actually happened — including the core retry/dead-letter path (a message redelivered N times and +/// then dead-lettered is directly assertable). +/// +/// var services = new ServiceCollection(); +/// services.AddFoundatio() +/// .Messaging.UseTestHarness() +/// .Messaging.AddHandler<OrderPlaced, SendConfirmationHandler>(); +/// // start hosted services, then: +/// await bus.PublishAsync(new OrderPlaced(42)); +/// await harness.WaitForIdleAsync(); +/// Assert.Single(harness.Published<OrderPlaced>()); +/// Assert.Empty(harness.DeadLetteredMessages); +/// +/// +public sealed class MessagingTestHarness : IAsyncDisposable +{ + private static readonly TimeSpan DefaultIdleTimeout = TimeSpan.FromSeconds(30); + + private readonly RecordingMessageTransport _transport; + private readonly ISerializer _serializer; + private readonly IMessageTypeRegistry _typeRegistry; + + public MessagingTestHarness(ISerializer? serializer = null, IMessageTypeRegistry? typeRegistry = null, TimeProvider? timeProvider = null) + { + _serializer = serializer ?? DefaultSerializer.Instance; + _typeRegistry = typeRegistry ?? new MessageTypeRegistry(); + _transport = new RecordingMessageTransport(timeProvider); + } + + /// The transport to run the bus over (an in-memory transport that records every movement). + public IMessageTransport Transport => _transport; + + /// Every message accepted by a queue-role send (a command on its way to one handler). + public IReadOnlyList SentMessages => _transport.Sent; + + /// Every message accepted by a topic-role send (an event on its way to each subscriber). + public IReadOnlyList PublishedMessages => _transport.Published; + + /// Every delivered message that settled as completed (handled successfully or auto-acked). + public IReadOnlyList HandledMessages => _transport.Handled; + + /// Every delivered message returned for redelivery (a retry). + public IReadOnlyList AbandonedMessages => _transport.Abandoned; + + /// + /// Every delivered message that settled terminally into the dead-letter sink. Records deaths made through the + /// transport API (the core's retry-exhausted/unrecoverable path); a broker-internal death — such as a message + /// whose TimeToLive lapsed before delivery — is visible in destination stats and ReceiveDeadLetteredAsync but + /// not recorded here. + /// + public IReadOnlyList DeadLetteredMessages => _transport.DeadLettered; + + /// The sent (queue-role) messages of type , deserialized. + public IReadOnlyList Sent() where T : class => Deserialize(_transport.Sent); + + /// The published (topic-role) messages of type , deserialized. + public IReadOnlyList Published() where T : class => Deserialize(_transport.Published); + + /// The successfully handled messages of type , deserialized. + public IReadOnlyList Handled() where T : class => Deserialize(_transport.Handled); + + /// The retried (abandoned for redelivery) messages of type , deserialized. + public IReadOnlyList Abandoned() where T : class => Deserialize(_transport.Abandoned); + + /// The dead-lettered messages of type , deserialized. + public IReadOnlyList DeadLettered() where T : class => Deserialize(_transport.DeadLettered); + + /// + /// Waits until the transport is quiescent — every known destination has nothing queued and nothing in flight — + /// so assertions observe the final state. Returns quickly when already idle (fast negative assertions). Throws + /// naming the still-busy destinations when the timeout (default 30s) lapses; + /// waits until idle or cancellation. + /// Store-parked work (delayed sends / delayed retries through a runtime store) is not transport activity; drain + /// it explicitly via the job schedule processor before waiting. + /// + public async Task WaitForIdleAsync(TimeSpan? timeout = null, CancellationToken cancellationToken = default) + { + var effectiveTimeout = timeout ?? DefaultIdleTimeout; + if (effectiveTimeout < TimeSpan.Zero && effectiveTimeout != Timeout.InfiniteTimeSpan) + throw new ArgumentOutOfRangeException(nameof(timeout), timeout, "Timeout must be non-negative or Timeout.InfiniteTimeSpan."); + + long deadline = effectiveTimeout == Timeout.InfiniteTimeSpan + ? Int64.MaxValue + : Environment.TickCount64 + (long)effectiveTimeout.TotalMilliseconds; + int stableChecks = 0; + + while (true) + { + cancellationToken.ThrowIfCancellationRequested(); + var pending = await _transport.GetPendingAsync(cancellationToken).ConfigureAwait(false); + + if (pending.Count == 0) + { + // Require two consecutive idle observations: a settling message can synchronously cascade into a new + // send, which a single snapshot could miss. + if (++stableChecks >= 2) + return; + } + else + { + stableChecks = 0; + + if (Environment.TickCount64 >= deadline) + { + var detail = new StringBuilder("The message bus did not become idle in time. Still busy: "); + detail.AppendJoin(", ", pending.Select(p => $"{p.Name} (queued={p.Queued}, working={p.Working})")); + throw new TimeoutException(detail.ToString()); + } + } + + await Task.Delay(25, cancellationToken).ConfigureAwait(false); + } + } + + public ValueTask DisposeAsync() => _transport.DisposeAsync(); + + private IReadOnlyList Deserialize(IReadOnlyList recordings) where T : class + { + string typeName = _typeRegistry.GetName(typeof(T)); + return recordings + .Where(r => String.Equals(r.MessageType, typeName, StringComparison.Ordinal)) + .Select(r => _serializer.Deserialize(r.Body, typeof(T)) as T) + .Where(m => m is not null) + .Select(m => m!) + .ToList(); + } +} diff --git a/src/Foundatio.Testing/RecordingMessageTransport.cs b/src/Foundatio.Testing/RecordingMessageTransport.cs new file mode 100644 index 000000000..196c95a04 --- /dev/null +++ b/src/Foundatio.Testing/RecordingMessageTransport.cs @@ -0,0 +1,186 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; + +namespace Foundatio.Messaging.Testing; + +/// +/// The harness transport: a fully-capable in-memory transport that records every send and settlement so tests can +/// assert on what actually moved through the bus, and tracks the destinations/sources it has seen so +/// can detect quiescence. +/// +internal sealed class RecordingMessageTransport : IMessageTransport, ISupportsPull, ISupportsPush, ISupportsVisibilityTimeout, + ISupportsDeadLetter, ISupportsRedeliveryDelay, ISupportsLockRenewal, ISupportsStats, + ISupportsProvisioning, ITransportInfo +{ + // A delayed redelivery lives only in the inner transport's timer until it fires — neither queued nor in flight — + // so idle detection would report quiescent while a retry is pending. Give the timer this long past its due time to + // materialize the redelivered message back into stats before the pending marker is dropped. + private static readonly TimeSpan _redeliveryGrace = TimeSpan.FromMilliseconds(250); + + private readonly InMemoryMessageTransport _inner; + private readonly TimeProvider _timeProvider; + private readonly ConcurrentQueue _sent = new(); + private readonly ConcurrentQueue _published = new(); + private readonly ConcurrentQueue _handled = new(); + private readonly ConcurrentQueue _abandoned = new(); + private readonly ConcurrentQueue _deadLettered = new(); + private readonly ConcurrentDictionary _knownNames = new(); + private readonly ConcurrentDictionary _pendingRedeliveries = new(); + + public RecordingMessageTransport(TimeProvider? timeProvider = null) + { + _timeProvider = timeProvider ?? TimeProvider.System; + _inner = new InMemoryMessageTransport(timeProvider); + } + + public IReadOnlyList Sent => [.. _sent]; + public IReadOnlyList Published => [.. _published]; + public IReadOnlyList Handled => [.. _handled]; + public IReadOnlyList Abandoned => [.. _abandoned]; + public IReadOnlyList DeadLettered => [.. _deadLettered]; + + public DeliveryGuarantee DeliveryGuarantee => _inner.DeliveryGuarantee; + public IReadOnlySet SupportedRoles => _inner.SupportedRoles; + public TransportCapabilities GetCapabilities(DestinationRole role) => _inner.GetCapabilities(role); + public TimeSpan? MaxVisibilityTimeout => _inner.MaxVisibilityTimeout; + public TimeSpan? MaxRedeliveryDelay => _inner.MaxRedeliveryDelay; + + public async Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + var result = await _inner.SendAsync(destination, messages, options, ct).ConfigureAwait(false); + + _knownNames.TryAdd(destination, 0); + var recordings = destination.Role == DestinationRole.Topic ? _published : _sent; + foreach (var message in messages) + { + recordings.Enqueue(new RecordedMessage + { + Destination = destination.Key, + Role = destination.Role, + MessageType = message.Headers.GetValueOrDefault(KnownHeaders.MessageType), + Body = message.Body, + Headers = message.Headers + }); + } + + return result; + } + + public Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct = default) + { + _knownNames.TryAdd(source, 0); + return _inner.ReceiveAsync(source, request, ct); + } + + public Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, TimeSpan visibility, CancellationToken ct = default) + { + _knownNames.TryAdd(source, 0); + return _inner.ReceiveAsync(source, request, visibility, ct); + } + + public Task SubscribeAsync(DestinationAddress source, Func onMessage, PushOptions options, CancellationToken ct = default) + { + _knownNames.TryAdd(source, 0); + return _inner.SubscribeAsync(source, onMessage, options, ct); + } + + public async Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) + { + await _inner.CompleteAsync(entry, ct).ConfigureAwait(false); + _handled.Enqueue(Record(entry)); + } + + public async Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) + { + await _inner.AbandonAsync(entry, ct).ConfigureAwait(false); + _abandoned.Enqueue(Record(entry)); + } + + public async Task AbandonAsync(TransportEntry entry, TimeSpan redeliveryDelay, CancellationToken ct = default) + { + Guid pendingToken = Guid.NewGuid(); + if (redeliveryDelay > TimeSpan.Zero) + _pendingRedeliveries[pendingToken] = (entry.Destination, _timeProvider.GetUtcNow().Add(redeliveryDelay)); + + try + { + await _inner.AbandonAsync(entry, redeliveryDelay, ct).ConfigureAwait(false); + } + catch + { + _pendingRedeliveries.TryRemove(pendingToken, out _); + throw; + } + + _abandoned.Enqueue(Record(entry)); + } + + public async Task DeadLetterAsync(TransportEntry entry, string? reason, CancellationToken ct = default) + { + await _inner.DeadLetterAsync(entry, reason, ct).ConfigureAwait(false); + _deadLettered.Enqueue(Record(entry) with { Reason = reason }); + } + + public Task> ReceiveDeadLetteredAsync(DestinationAddress destination, ReceiveRequest request, CancellationToken ct = default) + => _inner.ReceiveDeadLetteredAsync(destination, request, ct); + + public Task RenewLockAsync(TransportEntry entry, TimeSpan? duration, CancellationToken ct = default) + => _inner.RenewLockAsync(entry, duration, ct); + + public Task GetStatsAsync(DestinationAddress destination, CancellationToken ct = default) + => _inner.GetStatsAsync(destination, ct); + + public Task EnsureAsync(IReadOnlyList declarations, CancellationToken ct = default) + { + foreach (var declaration in declarations) + _knownNames.TryAdd(declaration.Address, 0); + return _inner.EnsureAsync(declarations, ct); + } + + public Task DeleteAsync(DestinationAddress destination, CancellationToken ct = default) => _inner.DeleteAsync(destination, ct); + + public Task ExistsAsync(DestinationAddress destination, CancellationToken ct = default) => _inner.ExistsAsync(destination, ct); + + public ValueTask DisposeAsync() => _inner.DisposeAsync(); + + // Aggregate pending work across every destination/source this transport has seen; idle means nothing queued, + // nothing in flight, and no delayed redelivery still waiting on its timer. + public async Task> GetPendingAsync(CancellationToken ct = default) + { + var now = _timeProvider.GetUtcNow(); + var scheduled = new Dictionary(); + foreach (var redelivery in _pendingRedeliveries) + { + if (now >= redelivery.Value.DueAt + _redeliveryGrace) + _pendingRedeliveries.TryRemove(redelivery.Key, out _); + else + scheduled[redelivery.Value.Destination] = scheduled.GetValueOrDefault(redelivery.Value.Destination) + 1; + } + + var pending = new List<(string, long, long)>(); + foreach (var address in _knownNames.Keys.OrderBy(a => a.Key, StringComparer.Ordinal)) + { + var stats = await _inner.GetStatsAsync(address, ct).ConfigureAwait(false); + long queued = stats.Queued + scheduled.GetValueOrDefault(address); + if (queued > 0 || stats.Working > 0) + pending.Add((address.Key, queued, stats.Working)); + } + + return pending; + } + + private static RecordedMessage Record(TransportEntry entry) => new() + { + Destination = entry.Destination.Key, + Role = DestinationRole.Queue, + MessageType = entry.Headers.GetValueOrDefault(KnownHeaders.MessageType), + Body = entry.Body, + Headers = entry.Headers, + Attempts = entry.DeliveryCount + }; +} diff --git a/src/Foundatio.Testing/TestingFoundatioBuilderExtensions.cs b/src/Foundatio.Testing/TestingFoundatioBuilderExtensions.cs new file mode 100644 index 000000000..cb899d6f2 --- /dev/null +++ b/src/Foundatio.Testing/TestingFoundatioBuilderExtensions.cs @@ -0,0 +1,26 @@ +using System; +using Foundatio.Messaging; +using Foundatio.Messaging.Testing; +using Foundatio.Serializer; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; + +namespace Foundatio; + +public static class TestingFoundatioBuilderExtensions +{ + /// + /// Runs messaging over a recording in-memory transport for tests. Resolve from + /// the container to await quiescence () and assert on the + /// messages that were sent, published, handled, retried, or dead-lettered. + /// + public static FoundatioBuilder UseTestHarness(this FoundatioBuilder.MessagingBuilder builder) + { + var services = ((IFoundatioBuilder)builder).Services; + services.TryAddSingleton(sp => new MessagingTestHarness( + sp.GetService(), + sp.GetService(), + sp.GetService())); + return builder.UseTransport(sp => sp.GetRequiredService().Transport); + } +} diff --git a/src/Foundatio.Xunit.v3/Logging/TestLoggerBase.cs b/src/Foundatio.Xunit.v3/Logging/TestLoggerBase.cs index 58597e3bd..b675ce80b 100644 --- a/src/Foundatio.Xunit.v3/Logging/TestLoggerBase.cs +++ b/src/Foundatio.Xunit.v3/Logging/TestLoggerBase.cs @@ -27,7 +27,7 @@ protected TestLoggerBase(ITestOutputHelper output, TestLoggerFixture fixture) /// /// Gets a cancellation token that is cancelled when the current test completes or /// when the test run is aborted/timed out. Pass this token to - /// + /// /// and other async operations to ensure automatic cleanup between tests. /// /// diff --git a/src/Foundatio.Xunit.v3/Logging/TestWithLoggingBase.cs b/src/Foundatio.Xunit.v3/Logging/TestWithLoggingBase.cs index 9dc993a44..e96c6a8f4 100644 --- a/src/Foundatio.Xunit.v3/Logging/TestWithLoggingBase.cs +++ b/src/Foundatio.Xunit.v3/Logging/TestWithLoggingBase.cs @@ -22,7 +22,7 @@ protected TestWithLoggingBase(ITestOutputHelper output) /// /// Gets a cancellation token that is cancelled when the current test completes or /// when the test run is aborted/timed out. Pass this token to - /// + /// /// and other async operations to ensure automatic cleanup between tests. /// /// diff --git a/src/Foundatio.Xunit/Logging/TestLoggerBase.cs b/src/Foundatio.Xunit/Logging/TestLoggerBase.cs index 2e1b29c02..9adf3d4b4 100644 --- a/src/Foundatio.Xunit/Logging/TestLoggerBase.cs +++ b/src/Foundatio.Xunit/Logging/TestLoggerBase.cs @@ -26,7 +26,7 @@ protected TestLoggerBase(ITestOutputHelper output, TestLoggerFixture fixture) /// /// Gets a cancellation token that is cancelled when the current test completes. - /// Pass this token to + /// Pass this token to /// and other async operations to ensure automatic cleanup between tests. /// /// diff --git a/src/Foundatio.Xunit/Logging/TestWithLoggingBase.cs b/src/Foundatio.Xunit/Logging/TestWithLoggingBase.cs index 4189b92c0..c9afc52c9 100644 --- a/src/Foundatio.Xunit/Logging/TestWithLoggingBase.cs +++ b/src/Foundatio.Xunit/Logging/TestWithLoggingBase.cs @@ -21,7 +21,7 @@ protected TestWithLoggingBase(ITestOutputHelper output) /// /// Gets a cancellation token that is cancelled when the current test completes. - /// Pass this token to + /// Pass this token to /// and other async operations to ensure automatic cleanup between tests. /// /// diff --git a/src/Foundatio/Caching/HybridAwareCacheClient.cs b/src/Foundatio/Caching/HybridAwareCacheClient.cs index 81532fc98..4b41c9a1f 100644 --- a/src/Foundatio/Caching/HybridAwareCacheClient.cs +++ b/src/Foundatio/Caching/HybridAwareCacheClient.cs @@ -2,7 +2,7 @@ using System.Collections.Generic; using System.Linq; using System.Threading.Tasks; -using Foundatio.Messaging; +using Foundatio.Messaging.Legacy; using Foundatio.Resilience; using Foundatio.Utility; using Microsoft.Extensions.Logging; diff --git a/src/Foundatio/Caching/HybridCacheClient.cs b/src/Foundatio/Caching/HybridCacheClient.cs index e248b49e1..ce417e6c2 100644 --- a/src/Foundatio/Caching/HybridCacheClient.cs +++ b/src/Foundatio/Caching/HybridCacheClient.cs @@ -5,7 +5,7 @@ using System.Threading; using System.Threading.Tasks; using Foundatio.AsyncEx; -using Foundatio.Messaging; +using Foundatio.Messaging.Legacy; using Foundatio.Resilience; using Foundatio.Utility; using Microsoft.Extensions.Logging; diff --git a/src/Foundatio.Extensions.Hosting/Cronos/CalendarHelper.cs b/src/Foundatio/Cronos/CalendarHelper.cs similarity index 99% rename from src/Foundatio.Extensions.Hosting/Cronos/CalendarHelper.cs rename to src/Foundatio/Cronos/CalendarHelper.cs index c447b746b..57630deca 100644 --- a/src/Foundatio.Extensions.Hosting/Cronos/CalendarHelper.cs +++ b/src/Foundatio/Cronos/CalendarHelper.cs @@ -5,7 +5,7 @@ using System; using System.Runtime.CompilerServices; -namespace Foundatio.Extensions.Hosting.Cronos; +namespace Foundatio.Cronos; internal static class CalendarHelper { diff --git a/src/Foundatio.Extensions.Hosting/Cronos/CronExpression.cs b/src/Foundatio/Cronos/CronExpression.cs similarity index 99% rename from src/Foundatio.Extensions.Hosting/Cronos/CronExpression.cs rename to src/Foundatio/Cronos/CronExpression.cs index 9068ed36d..0518952e3 100644 --- a/src/Foundatio.Extensions.Hosting/Cronos/CronExpression.cs +++ b/src/Foundatio/Cronos/CronExpression.cs @@ -26,7 +26,7 @@ using System.Runtime.CompilerServices; using System.Text; -namespace Foundatio.Extensions.Hosting.Cronos; +namespace Foundatio.Cronos; /// /// Provides a parser and scheduler for cron expressions. diff --git a/src/Foundatio.Extensions.Hosting/Cronos/CronExpressionFlag.cs b/src/Foundatio/Cronos/CronExpressionFlag.cs similarity index 96% rename from src/Foundatio.Extensions.Hosting/Cronos/CronExpressionFlag.cs rename to src/Foundatio/Cronos/CronExpressionFlag.cs index df7b2fc7e..74b7bba04 100644 --- a/src/Foundatio.Extensions.Hosting/Cronos/CronExpressionFlag.cs +++ b/src/Foundatio/Cronos/CronExpressionFlag.cs @@ -22,7 +22,7 @@ using System; -namespace Foundatio.Extensions.Hosting.Cronos; +namespace Foundatio.Cronos; [Flags] internal enum CronExpressionFlag : byte diff --git a/src/Foundatio.Extensions.Hosting/Cronos/CronField.cs b/src/Foundatio/Cronos/CronField.cs similarity index 98% rename from src/Foundatio.Extensions.Hosting/Cronos/CronField.cs rename to src/Foundatio/Cronos/CronField.cs index 44c9fe8ef..5065f0a01 100644 --- a/src/Foundatio.Extensions.Hosting/Cronos/CronField.cs +++ b/src/Foundatio/Cronos/CronField.cs @@ -20,7 +20,7 @@ // OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE // SOFTWARE. -namespace Foundatio.Extensions.Hosting.Cronos; +namespace Foundatio.Cronos; internal sealed class CronField { diff --git a/src/Foundatio.Extensions.Hosting/Cronos/CronFormat.cs b/src/Foundatio/Cronos/CronFormat.cs similarity index 97% rename from src/Foundatio.Extensions.Hosting/Cronos/CronFormat.cs rename to src/Foundatio/Cronos/CronFormat.cs index 2e762e5e6..776e2684a 100644 --- a/src/Foundatio.Extensions.Hosting/Cronos/CronFormat.cs +++ b/src/Foundatio/Cronos/CronFormat.cs @@ -22,7 +22,7 @@ using System; -namespace Foundatio.Extensions.Hosting.Cronos; +namespace Foundatio.Cronos; /// /// Defines the cron format options that customize string parsing for . diff --git a/src/Foundatio.Extensions.Hosting/Cronos/CronFormatException.cs b/src/Foundatio/Cronos/CronFormatException.cs similarity index 97% rename from src/Foundatio.Extensions.Hosting/Cronos/CronFormatException.cs rename to src/Foundatio/Cronos/CronFormatException.cs index 7df867527..fe13770a8 100644 --- a/src/Foundatio.Extensions.Hosting/Cronos/CronFormatException.cs +++ b/src/Foundatio/Cronos/CronFormatException.cs @@ -22,7 +22,7 @@ using System; -namespace Foundatio.Extensions.Hosting.Cronos; +namespace Foundatio.Cronos; /// /// Represents an exception that's thrown, when invalid Cron expression is given. diff --git a/src/Foundatio.Extensions.Hosting/Cronos/TimeZoneHelper.cs b/src/Foundatio/Cronos/TimeZoneHelper.cs similarity index 99% rename from src/Foundatio.Extensions.Hosting/Cronos/TimeZoneHelper.cs rename to src/Foundatio/Cronos/TimeZoneHelper.cs index 587cfa248..4eaef9ef0 100644 --- a/src/Foundatio.Extensions.Hosting/Cronos/TimeZoneHelper.cs +++ b/src/Foundatio/Cronos/TimeZoneHelper.cs @@ -22,7 +22,7 @@ using System; -namespace Foundatio.Extensions.Hosting.Cronos; +namespace Foundatio.Cronos; internal static class TimeZoneHelper { diff --git a/src/Foundatio/Foundatio.csproj b/src/Foundatio/Foundatio.csproj index 04a1ec8d7..3fb6df955 100644 --- a/src/Foundatio/Foundatio.csproj +++ b/src/Foundatio/Foundatio.csproj @@ -1,9 +1,16 @@ + + + true + + + diff --git a/src/Foundatio/FoundatioServicesExtensions.cs b/src/Foundatio/FoundatioServicesExtensions.cs index 07b184b78..53abffdc3 100644 --- a/src/Foundatio/FoundatioServicesExtensions.cs +++ b/src/Foundatio/FoundatioServicesExtensions.cs @@ -1,13 +1,20 @@ using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; using Foundatio.Caching; using Foundatio.Extensions; +using Foundatio.Jobs; using Foundatio.Lock; using Foundatio.Messaging; +using Legacy = Foundatio.Messaging.Legacy; using Foundatio.Queues; using Foundatio.Resilience; using Foundatio.Serializer; using Foundatio.Storage; using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Logging; namespace Foundatio; @@ -36,6 +43,7 @@ internal FoundatioBuilder(IServiceCollection services) Storage = new StorageBuilder(this); Messaging = new MessagingBuilder(this); Queueing = new QueueingBuilder(this); + Jobs = new JobsBuilder(this); Locking = new LockingBuilder(this); } @@ -62,6 +70,11 @@ internal FoundatioBuilder(IServiceCollection services) /// public QueueingBuilder Queueing { get; } + /// + /// Configure background job runtime services for Foundatio. + /// + public JobsBuilder Jobs { get; } + /// /// Configure locking services for Foundatio. /// @@ -237,6 +250,9 @@ public class MessagingBuilder : IFoundatioBuilder { private readonly FoundatioBuilder _builder; private readonly IServiceCollection _services; + private bool _routingServicesRegistered; + private bool _topologyServicesRegistered; + private TopologyMode _topologyMode = TopologyMode.Ensure; internal MessagingBuilder(IFoundatioBuilder builder) { @@ -244,40 +260,336 @@ internal MessagingBuilder(IFoundatioBuilder builder) _services = builder.Services; } + /// + /// Selects how the messaging client administers topology: creates missing + /// destinations on use and at handler-host startup (default), only checks + /// they exist and throws when missing, and never touches topology. + /// + public MessagingBuilder ConfigureTopology(TopologyMode mode) + { + _topologyMode = mode; + return this; + } + IServiceCollection IFoundatioBuilder.Services => _services; FoundatioBuilder IFoundatioBuilder.Builder => _builder; - public FoundatioBuilder Use(IMessageBus messageBus) + public FoundatioBuilder Use(Legacy.IMessageBus messageBus) { _services.ReplaceSingleton(_ => messageBus); - _services.ReplaceSingleton(sp => sp.GetRequiredService()); - _services.ReplaceSingleton(sp => sp.GetRequiredService()); + _services.ReplaceSingleton(sp => sp.GetRequiredService()); + _services.ReplaceSingleton(sp => sp.GetRequiredService()); + return _builder; + } + + public FoundatioBuilder Use(Func factory) + { + _services.ReplaceSingleton(factory); + _services.ReplaceSingleton(sp => sp.GetRequiredService()); + _services.ReplaceSingleton(sp => sp.GetRequiredService()); + return _builder; + } + + public MessagingBuilder ConfigureRouting(Action configure) + { + ArgumentNullException.ThrowIfNull(configure); + + _services.AddSingleton>(configure); + RegisterRoutingServices(); + return this; + } + + // The core owns retry and dead-letter behavior so it is identical across transports. This configures the + // default policy applied to queue and pub/sub consumers; a consumer can still override MaxAttempts/backoff. + public MessagingBuilder ConfigureRetry(RetryPolicy policy) + { + ArgumentNullException.ThrowIfNull(policy); + _services.ReplaceSingleton(_ => policy); + return this; + } + + public MessagingBuilder ConfigureRetry(Func configure) + { + ArgumentNullException.ThrowIfNull(configure); + return ConfigureRetry(configure(new RetryPolicy())); + } + + // Registers a stable wire name for a message type so the discriminator survives assembly/namespace moves and + // grouped/interface consumers can resolve and deserialize the concrete payload type. + public MessagingBuilder RegisterMessageType(string name) where T : class + { + ArgumentException.ThrowIfNullOrEmpty(name); + _services.AddSingleton(new MessageTypeRegistration(name, typeof(T))); + return this; + } + + public FoundatioBuilder UseInMemory(Legacy.InMemoryMessageBusOptions? options = null) + { + _services.ReplaceSingleton(sp => new Legacy.InMemoryMessageBus(options.UseServices(sp))); + _services.ReplaceSingleton(sp => sp.GetRequiredService()); + _services.ReplaceSingleton(sp => sp.GetRequiredService()); + RegisterMessagingRuntime(sp => new InMemoryMessageTransport(sp.GetService())); + return _builder; + } + + public FoundatioBuilder UseInMemory(Builder config) + { + _services.ReplaceSingleton(sp => new Legacy.InMemoryMessageBus(b => b.Configure(config).UseServices(sp))); + _services.ReplaceSingleton(sp => sp.GetRequiredService()); + _services.ReplaceSingleton(sp => sp.GetRequiredService()); + RegisterMessagingRuntime(sp => new InMemoryMessageTransport(sp.GetService())); + return _builder; + } + + public FoundatioBuilder UseTransport(IMessageTransport transport) + { + ArgumentNullException.ThrowIfNull(transport); + RegisterMessagingRuntime(_ => transport); + return _builder; + } + + public FoundatioBuilder UseTransport(Func factory) + { + RegisterMessagingRuntime(factory); + return _builder; + } + + /// + /// Registers a handler for messages of type . Registration carries no topology + /// decision — the caller's verb on decides delivery: a SendAsync is processed + /// by exactly one handler instance across the fleet (competing consumers), and a PublishAsync is received + /// once per subscribing service (a scaled service's instances compete), or by every instance when + /// is set. The handler is resolved from DI in its own scope + /// per message (so it can inject scoped dependencies); throwing triggers the retry/dead-letter policy. A single + /// hosted service starts and stops all registered handlers. + /// + public FoundatioBuilder AddHandler(Action? configure = null) + where TMessage : class where THandler : class, IMessageHandler + { + _services.TryAddScoped(); + // Each handler class is its own subscriber group ("{service}.{handler}"), so every handler registered for + // an event type receives its own copy of each published message. + return AddHandlerRegistration(typeof(THandler).Name, static (sp, message, ct) => DispatchAsync(sp, message, ct), + options => + { + configure?.Invoke(options); + options.SubscriptionQualifier ??= typeof(THandler).Name; + }); + } + + /// + /// Registers a delegate handler for messages of type ; see + /// for the delivery semantics. + /// + public FoundatioBuilder AddHandler(Func, CancellationToken, Task> handler, Action? configure = null) + where TMessage : class + { + ArgumentNullException.ThrowIfNull(handler); + return AddHandlerRegistration(null, (_, message, ct) => handler(message, ct), configure); + } + + private FoundatioBuilder AddHandlerRegistration(string? handlerName, Func, CancellationToken, Task> dispatch, Action? configure) + where TMessage : class + { + string suffix = handlerName is null ? String.Empty : $" -> {handlerName}"; + _services.AddSingleton(new MessageHandlerRegistration + { + Description = $"handler:{typeof(TMessage).Name}{suffix}", + StartAsync = async (sp, ct) => + { + var options = new MessageSubscriptionOptions(); + configure?.Invoke(options); + return await sp.GetRequiredService() + .SubscribeAsync((message, c) => dispatch(sp, message, c), options, ct).ConfigureAwait(false); + } + }); + + if (!_services.Any(s => s.ServiceType == typeof(IHostedService) && s.ImplementationType == typeof(MessageHandlerHostedService))) + _services.AddSingleton(); + return _builder; } - public FoundatioBuilder Use(Func factory) + private static async Task DispatchAsync(IServiceProvider serviceProvider, IMessageContext message, CancellationToken cancellationToken) + where TMessage : class where THandler : class, IMessageHandler + { + await using var scope = serviceProvider.CreateAsyncScope(); + var handler = scope.ServiceProvider.GetRequiredService(); + await handler.HandleAsync(message, cancellationToken).ConfigureAwait(false); + } + + private void RegisterMessagingRuntime(Func factory) { _services.ReplaceSingleton(factory); - _services.ReplaceSingleton(sp => sp.GetRequiredService()); - _services.ReplaceSingleton(sp => sp.GetRequiredService()); + // Resolved lazily so ConfigureTopology can be called before or after the Use* transport registration. + _services.ReplaceSingleton(_ => new MessagingTopologyOptions(_topologyMode)); + RegisterMessageTopology(); + RegisterMessageClients(); + } + + private void RegisterRoutingServices() + { + if (_routingServicesRegistered) + return; + + _routingServicesRegistered = true; + _services.ReplaceSingleton(sp => + { + var options = new MessageRoutingOptions(); + var builder = new MessageRoutingOptionsBuilder(options); + foreach (var configure in sp.GetServices>()) + configure(builder); + + return options; + }); + _services.ReplaceSingleton(sp => new DefaultMessageRouter(sp.GetRequiredService())); + } + + private void RegisterMessageTopology() + { + RegisterRoutingServices(); + + if (_topologyServicesRegistered) + return; + + _topologyServicesRegistered = true; + _services.ReplaceSingleton(sp => new MessageTopology( + sp.GetRequiredService(), + sp.GetRequiredService())); + } + + private void RegisterMessageClients() + { + RegisterRoutingServices(); + _services.ReplaceSingleton(sp => new MessageTypeRegistry(sp.GetServices())); + _services.ReplaceSingleton(sp => new MessageBus(sp.GetRequiredService(), new MessageBusOptions + { + Serializer = sp.GetService() ?? DefaultSerializer.Instance, + Router = sp.GetService() ?? DefaultMessageRouter.Instance, + MessageTypes = sp.GetService() ?? new MessageTypeRegistry(), + RuntimeStore = sp.GetService() ?? sp.GetService(), + RetryPolicy = sp.GetService() ?? new RetryPolicy(), + Topology = sp.GetService()?.Mode ?? TopologyMode.Ensure, + // The transport is a shared DI singleton owned by the container; the bus must not dispose it. + OwnsTransport = false, + TimeProvider = sp.GetService() ?? TimeProvider.System, + LoggerFactory = sp.GetService() + })); + } + } + + public class JobsBuilder : IFoundatioBuilder + { + private readonly FoundatioBuilder _builder; + private readonly IServiceCollection _services; + + internal JobsBuilder(IFoundatioBuilder builder) + { + _builder = builder.Builder; + _services = builder.Services; + } + + IServiceCollection IFoundatioBuilder.Services => _services; + FoundatioBuilder IFoundatioBuilder.Builder => _builder; + + public FoundatioBuilder UseRuntimeStore(IJobRuntimeStore store) + { + _services.ReplaceSingleton(_ => store); + RegisterJobServices(); + return _builder; + } + + public FoundatioBuilder UseRuntimeStore(Func factory) + { + _services.ReplaceSingleton(factory); + RegisterJobServices(); return _builder; } - public FoundatioBuilder UseInMemory(InMemoryMessageBusOptions? options = null) + public FoundatioBuilder UseInMemoryRuntime() { - _services.ReplaceSingleton(sp => new InMemoryMessageBus(options.UseServices(sp))); - _services.ReplaceSingleton(sp => sp.GetRequiredService()); - _services.ReplaceSingleton(sp => sp.GetRequiredService()); + _services.ReplaceSingleton(sp => new InMemoryJobRuntimeStore(sp.GetService())); + RegisterJobServices(); return _builder; } - public FoundatioBuilder UseInMemory(Builder config) + public FoundatioBuilder Register(string name) where TJob : IJob { - _services.ReplaceSingleton(sp => new InMemoryMessageBus(b => b.Configure(config).UseServices(sp))); - _services.ReplaceSingleton(sp => sp.GetRequiredService()); - _services.ReplaceSingleton(sp => sp.GetRequiredService()); + ArgumentException.ThrowIfNullOrEmpty(name); + _services.AddSingleton(new JobTypeRegistration(name, typeof(TJob))); return _builder; } + + /// + /// Registers a recurring (CRON) job. The schedule is materialized once into the shared runtime store per + /// occurrence, so decides fan-out (Global = one instance per tick, + /// PerNode = every instance per tick). Scheduled automatically when the runtime pump starts — no manual + /// call needed. Requires a runtime store ( + /// / ). + /// + public FoundatioBuilder AddCronJob(string cronSchedule, Action? configure = null) where TJob : IJob + { + ArgumentException.ThrowIfNullOrEmpty(cronSchedule); + + var options = new CronJobOptions(); + configure?.Invoke(options); + string name = options.Name ?? typeof(TJob).Name; + + _services.AddSingleton(new JobTypeRegistration(name, typeof(TJob))); + _services.AddSingleton(new ScheduledJobDefinition + { + Name = name, + Cron = cronSchedule, + JobType = typeof(TJob), + Scope = options.Scope, + Overlap = options.Overlap, + MisfireWindow = options.MisfireWindow, + MaxRetries = options.MaxRetries, + Enabled = options.Enabled, + TimeZone = options.TimeZone, + Arguments = options.Arguments + }); + return _builder; + } + + /// + /// Tunes the auto-registered runtime pump (cadence, batch size, or + /// to opt out of automatic pumping and take manual control). + /// + public FoundatioBuilder ConfigureRuntimePump(Action configure) + { + ArgumentNullException.ThrowIfNull(configure); + var options = new JobRuntimePumpOptions(); + configure(options); + _services.ReplaceSingleton(_ => options); + return _builder; + } + + private void RegisterJobServices() + { + _services.ReplaceSingleton(sp => new JobTypeRegistry(sp.GetServices())); + _services.ReplaceSingleton(sp => sp.GetRequiredService()); + _services.ReplaceSingleton(sp => new JobClient(sp.GetRequiredService(), sp.GetService(), sp.GetRequiredService(), sp.GetService())); + _services.ReplaceSingleton(sp => new JobWorker(sp.GetRequiredService(), sp, sp.GetService(), jobTypes: sp.GetRequiredService(), serializer: sp.GetService(), + maxConcurrency: sp.GetService()?.WorkerConcurrency ?? 1)); + _services.ReplaceSingleton(); + _services.ReplaceSingleton(sp => new JobScheduleProcessor( + sp.GetRequiredService(), + sp.GetRequiredService(), + sp.GetRequiredService(), + sp.GetService(), + transport: sp.GetService(), + jobTypes: sp.GetRequiredService(), + serializer: sp.GetService())); + + // A runtime store is inert without something draining it, so register the pump alongside the store: in a + // hosted process it runs jobs and the messaging delayed-delivery fallback automatically (no separate + // AddJobRuntimeService call); in a non-hosted process the IHostedService is simply never started. Guarded so + // repeated UseRuntimeStore/UseInMemoryRuntime calls don't stack multiple pumps. Options default unless + // AddJobRuntimeService (or a registered JobRuntimePumpOptions) overrides them. + if (!_services.Any(s => s.ServiceType == typeof(IHostedService) && s.ImplementationType == typeof(JobRuntimePumpService))) + _services.AddSingleton(); + } } public class QueueingBuilder : IFoundatioBuilder @@ -350,7 +662,7 @@ public FoundatioBuilder UseCache() // gets all services from the ICacheClient instance _services.ReplaceSingleton(sp => new CacheLockProvider( sp.GetRequiredService(), - sp.GetService(), // optional for more efficient lock release notifications + sp.GetService(), // optional for more efficient lock release notifications sp.GetService(), sp.GetService(), sp.GetService() diff --git a/src/Foundatio/Jobs/IJob.cs b/src/Foundatio/Jobs/IJob.cs index ac5cefd10..23ac3ace6 100644 --- a/src/Foundatio/Jobs/IJob.cs +++ b/src/Foundatio/Jobs/IJob.cs @@ -1,44 +1,34 @@ using System; -using System.Linq; -using System.Threading; using System.Threading.Tasks; using Foundatio.Utility; -using Microsoft.Extensions.Logging; namespace Foundatio.Jobs; /// -/// Represents a unit of background work that can be executed once or continuously. -/// Implement this interface to create custom jobs for scheduled tasks, queue processing, or maintenance operations. +/// Represents a unit of background work run by the durable job runtime. Every run is handed a +/// carrying its cancellation token, identity, attempt number, and store-backed +/// progress/heartbeat helpers — a job uses what it needs and ignores the rest. /// public interface IJob { /// /// Executes the job's work. /// - /// Token to signal that the job should stop. + /// The execution context for this run (cancellation, identity, progress, heartbeat). /// A result indicating success, failure, or cancellation. - Task RunAsync(CancellationToken cancellationToken = default); + Task RunAsync(JobExecutionContext context); } -/// -/// A job that exposes configurable options for execution behavior. -/// -public interface IJobWithOptions : IJob +public static class JobExtensions { /// - /// Gets or sets the options controlling job execution (name, interval, iteration limit). + /// Runs the job, converting cancellation and unhandled exceptions into a instead of throwing. /// - JobOptions? Options { get; set; } -} - -public static class JobExtensions -{ - public static async Task TryRunAsync(this IJob job, CancellationToken cancellationToken = default) + public static async Task TryRunAsync(this IJob job, JobExecutionContext context) { try { - return await job.RunAsync(cancellationToken).AnyContext(); + return await job.RunAsync(context).AnyContext(); } catch (OperationCanceledException) { @@ -49,97 +39,4 @@ public static async Task TryRunAsync(this IJob job, CancellationToken return JobResult.FromException(ex); } } - - /// - /// Runs the job continuously until the cancellation token is set or the iteration limit is reached. - /// - /// Returns the iteration count for normal jobs. For queue-based jobs this will be the number of items processed successfully. - public static Task RunContinuousAsync(this IJob job, TimeSpan? interval = null, int iterationLimit = -1, - CancellationToken cancellationToken = default, Func>? continuationCallback = null) - { - var options = JobOptions.GetDefaults(job); - options.Interval = interval; - options.IterationLimit = iterationLimit; - return RunContinuousAsync(job, options, cancellationToken, continuationCallback); - } - - /// - /// Runs the job continuously until the cancellation token is set or the iteration limit is reached. - /// - /// Returns the iteration count for normal jobs. For queue based jobs this will be the amount of items processed successfully. - public static async Task RunContinuousAsync(this IJob job, JobOptions options, CancellationToken cancellationToken = default, Func>? continuationCallback = null) - { - int iterations = 0; - var logger = job.GetLogger(); - - int queueItemsProcessed = 0; - bool isQueueJob = job.GetType().GetInterfaces().Any(x => x.IsGenericType && x.GetGenericTypeDefinition() == typeof(IQueueJob<>)); - - string jobId = Guid.NewGuid().ToString("N").Substring(0, 10); - using var jobScope = logger.BeginScope(s => s.Property("job.name", options.Name ?? String.Empty).Property("job.id", jobId)); - logger.LogInformation("Starting continuous job type {JobName} on machine {MachineName}...", options.Name, Environment.MachineName); - - while (!cancellationToken.IsCancellationRequested) - { - using var activity = FoundatioDiagnostics.ActivitySource.StartActivity($"Job: {options.Name}"); - - string jobRunId = Guid.NewGuid().ToString("N").Substring(0, 10); - using var _ = logger.BeginScope(s => s.Property("job.run_id", jobRunId)); - var result = await job.TryRunAsync(cancellationToken).AnyContext(); - logger.LogJobResult(result, options.Name); - - iterations++; - if (isQueueJob && result.IsSuccess) - queueItemsProcessed++; - - if (cancellationToken.IsCancellationRequested || (options.IterationLimit > -1 && options.IterationLimit <= iterations)) - break; - - if (result.Error != null) - { - await job.GetTimeProvider().SafeDelay(TimeSpan.FromMilliseconds(Math.Max((int)(options.Interval?.TotalMilliseconds ?? 0), 100)), cancellationToken).AnyContext(); - } - else if (options.Interval.HasValue && options.Interval.Value > TimeSpan.Zero) - { - await job.GetTimeProvider().SafeDelay(options.Interval.Value, cancellationToken).AnyContext(); - } - - // needed to yield back a task for jobs that aren't async - await Task.Yield(); - - if (cancellationToken.IsCancellationRequested) - break; - - if (continuationCallback is null) - continue; - - try - { - if (!await continuationCallback().AnyContext()) - break; - } - catch (Exception ex) - { - logger.LogError(ex, "Error in continuation callback: {Message}", ex.Message); - } - } - - if (cancellationToken.IsCancellationRequested) - logger.LogTrace("Job cancellation requested"); - - if (options.IterationLimit > 0) - { - logger.LogInformation( - "Stopping continuous job type {JobName} on machine {MachineName}: Job ran {Iterations} times (Limit={IterationLimit})", - options.Name, Environment.MachineName, iterations, options.IterationLimit); - } - else - { - logger.LogInformation( - "Stopping continuous job type {JobName} on machine {MachineName}: Job ran {Iterations} times", - options.Name, Environment.MachineName, iterations); - } - - return isQueueJob ? queueItemsProcessed : iterations; - } } diff --git a/src/Foundatio/Jobs/IQueueJob.cs b/src/Foundatio/Jobs/IQueueJob.cs index 2f396e3b2..ef7eafb76 100644 --- a/src/Foundatio/Jobs/IQueueJob.cs +++ b/src/Foundatio/Jobs/IQueueJob.cs @@ -5,7 +5,7 @@ using Foundatio.Utility; using Microsoft.Extensions.Logging; -namespace Foundatio.Jobs; +namespace Foundatio.Jobs.Legacy; /// /// A job that processes items from a queue. Each invocation of diff --git a/src/Foundatio/Jobs/JobAttribute.cs b/src/Foundatio/Jobs/JobAttribute.cs index 8b32f5ab4..b39b99fdc 100644 --- a/src/Foundatio/Jobs/JobAttribute.cs +++ b/src/Foundatio/Jobs/JobAttribute.cs @@ -1,6 +1,6 @@ using System; -namespace Foundatio.Jobs; +namespace Foundatio.Jobs.Legacy; [AttributeUsage(AttributeTargets.Class, AllowMultiple = false, Inherited = true)] public class JobAttribute : Attribute diff --git a/src/Foundatio/Jobs/JobBase.cs b/src/Foundatio/Jobs/JobBase.cs index ead6e3b48..e5b4711ec 100644 --- a/src/Foundatio/Jobs/JobBase.cs +++ b/src/Foundatio/Jobs/JobBase.cs @@ -6,7 +6,7 @@ using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; -namespace Foundatio.Jobs; +namespace Foundatio.Jobs.Legacy; public abstract class JobBase : IJob, IHaveLogger, IHaveLoggerFactory, IHaveTimeProvider, IHaveResiliencePolicyProvider { diff --git a/src/Foundatio/Jobs/JobContext.cs b/src/Foundatio/Jobs/JobContext.cs index c955f2657..2550bbdd6 100644 --- a/src/Foundatio/Jobs/JobContext.cs +++ b/src/Foundatio/Jobs/JobContext.cs @@ -2,7 +2,7 @@ using System.Threading.Tasks; using Foundatio.Lock; -namespace Foundatio.Jobs; +namespace Foundatio.Jobs.Legacy; public class JobContext { diff --git a/src/Foundatio/Jobs/JobOptions.cs b/src/Foundatio/Jobs/JobOptions.cs index 8960a0fbd..c0d175687 100644 --- a/src/Foundatio/Jobs/JobOptions.cs +++ b/src/Foundatio/Jobs/JobOptions.cs @@ -3,7 +3,7 @@ using Foundatio.Extensions; using Foundatio.Utility; -namespace Foundatio.Jobs; +namespace Foundatio.Jobs.Legacy; public class JobOptions { diff --git a/src/Foundatio/Jobs/JobRunner.cs b/src/Foundatio/Jobs/JobRunner.cs index 0437123f4..c7f121028 100644 --- a/src/Foundatio/Jobs/JobRunner.cs +++ b/src/Foundatio/Jobs/JobRunner.cs @@ -9,7 +9,7 @@ using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; -namespace Foundatio.Jobs; +namespace Foundatio.Jobs.Legacy; public class JobRunner { diff --git a/src/Foundatio/Jobs/JobRuntime.cs b/src/Foundatio/Jobs/JobRuntime.cs new file mode 100644 index 000000000..d9bb8be69 --- /dev/null +++ b/src/Foundatio/Jobs/JobRuntime.cs @@ -0,0 +1,1150 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Diagnostics.Metrics; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Foundatio.Serializer; +using Foundatio.Utility; +using Microsoft.Extensions.DependencyInjection; + +namespace Foundatio.Jobs; + +/// +/// Core-owned durable-job instruments, shared by every so job throughput and run latency are +/// observable independent of the runtime store implementation. +/// +internal static class JobInstruments +{ + public static readonly Counter Started = FoundatioDiagnostics.Meter.CreateCounter("foundatio.jobs.started", description: "Number of durable jobs started"); + public static readonly Counter Completed = FoundatioDiagnostics.Meter.CreateCounter("foundatio.jobs.completed", description: "Number of durable jobs completed successfully"); + public static readonly Counter Failed = FoundatioDiagnostics.Meter.CreateCounter("foundatio.jobs.failed", description: "Number of durable jobs that failed"); + public static readonly Counter Cancelled = FoundatioDiagnostics.Meter.CreateCounter("foundatio.jobs.cancelled", description: "Number of durable jobs that were cancelled"); + public static readonly Histogram RunTime = FoundatioDiagnostics.Meter.CreateHistogram("foundatio.jobs.runtime", unit: "ms", description: "Durable job execution time"); +} + +public enum JobStatus +{ + Queued, + Scheduled, + Processing, + Completed, + Failed, + Cancelled, + DeadLettered +} + +public enum ScheduledDispatchKind +{ + QueueMessage, + PubSubMessage, + JobOccurrence +} + +public sealed record JobState +{ + public required string JobId { get; init; } + public required string Name { get; init; } + public string? JobType { get; init; } + + /// Serialized per-invocation arguments (see ); null when the job takes none. + public ReadOnlyMemory? Payload { get; init; } + + /// Discriminator for the payload type (the argument type's full name), stored for forensics and mismatch diagnostics. + public string? PayloadType { get; init; } + public JobStatus Status { get; init; } = JobStatus.Queued; + public int? Progress { get; init; } + public string? ProgressMessage { get; init; } + public int Attempt { get; init; } + public string? NodeId { get; init; } + public DateTimeOffset CreatedUtc { get; init; } = DateTimeOffset.UtcNow; + public DateTimeOffset LastUpdatedUtc { get; init; } = DateTimeOffset.UtcNow; + public DateTimeOffset? StartedUtc { get; init; } + public DateTimeOffset? CompletedUtc { get; init; } + public DateTimeOffset? LeaseExpiresUtc { get; init; } + public string? Error { get; init; } + public bool CancellationRequested { get; init; } + public DateTimeOffset? ScheduledForUtc { get; init; } +} + +public sealed record JobStatePatch +{ + public JobStatus? Status { get; init; } + public string? JobType { get; init; } + public int? Progress { get; init; } + public string? ProgressMessage { get; init; } + public string? Error { get; init; } + public int AttemptDelta { get; init; } + public string? NodeId { get; init; } + public bool ClearNodeId { get; init; } + public DateTimeOffset? LeaseExpiresUtc { get; init; } + public bool ClearLeaseExpiresUtc { get; init; } + public DateTimeOffset? LastUpdatedUtc { get; init; } + public DateTimeOffset? StartedUtc { get; init; } + public DateTimeOffset? CompletedUtc { get; init; } + public bool? CancellationRequested { get; init; } +} + +public sealed record JobQuery +{ + public string? Name { get; init; } + public JobStatus? Status { get; init; } + public int Limit { get; init; } = 100; + + /// + /// When true, CRON occurrences (jobs with set) are excluded. The job + /// scheduler is the sole executor of occurrences, so the generic worker must not claim them — otherwise it would + /// run them without the per-definition retry/dead-letter accounting that lives in the scheduler. + /// + public bool ExcludeOccurrences { get; init; } +} + +public sealed record ScheduledDispatchState +{ + public required string DispatchId { get; init; } + public ScheduledDispatchKind Kind { get; init; } + + /// The transport destination for queue/pub-sub message dispatches; null for job occurrences. + public DestinationAddress? Destination { get; init; } + + /// The scheduled job definition name for dispatches; null for message dispatches. + public string? JobName { get; init; } + + public required ReadOnlyMemory Body { get; init; } + public MessageHeaders Headers { get; init; } = MessageHeaders.Empty; + public TransportSendOptions Options { get; init; } = new(); + public DateTimeOffset DueUtc { get; init; } + public string? ClaimOwner { get; init; } + public DateTimeOffset? ClaimExpiresUtc { get; init; } + public int Attempts { get; init; } + public string? JobId { get; init; } +} + +public sealed record JobRequestOptions +{ + public string? JobId { get; init; } + public string? Name { get; init; } +} + +public sealed record JobTypeRegistration(string Name, Type JobType); + +public interface IJobTypeRegistry +{ + string GetName(Type jobType); + Type Resolve(string name); +} + +public sealed class JobTypeRegistry : IJobTypeRegistry +{ + private readonly Dictionary _nameToType; + private readonly Dictionary _typeToName; + + public JobTypeRegistry(IEnumerable? registrations = null) + { + _nameToType = new Dictionary(StringComparer.Ordinal); + _typeToName = new Dictionary(); + + foreach (var registration in registrations ?? []) + Add(registration); + } + + public string GetName(Type jobType) + { + ArgumentNullException.ThrowIfNull(jobType); + if (!typeof(IJob).IsAssignableFrom(jobType)) + throw new ArgumentException("Job type must implement IJob.", nameof(jobType)); + + return _typeToName.TryGetValue(jobType, out string? name) + ? name + : jobType.FullName ?? jobType.Name; + } + + public Type Resolve(string name) + { + ArgumentException.ThrowIfNullOrEmpty(name); + + if (_nameToType.TryGetValue(name, out var registered)) + return registered; + + var jobType = Type.GetType(name, throwOnError: false); + if (jobType is null) + { + foreach (var assembly in AppDomain.CurrentDomain.GetAssemblies()) + { + jobType = assembly.GetType(name, throwOnError: false); + if (jobType is not null) + break; + } + } + + if (jobType is null || !typeof(IJob).IsAssignableFrom(jobType)) + throw new InvalidOperationException($"Job type \"{name}\" could not be resolved to an IJob implementation."); + + return jobType; + } + + private void Add(JobTypeRegistration registration) + { + ArgumentNullException.ThrowIfNull(registration); + ArgumentException.ThrowIfNullOrEmpty(registration.Name); + ArgumentNullException.ThrowIfNull(registration.JobType); + + if (!typeof(IJob).IsAssignableFrom(registration.JobType)) + throw new ArgumentException("Job type must implement IJob.", nameof(registration)); + + if (_nameToType.TryGetValue(registration.Name, out var existing) && existing != registration.JobType) + throw new InvalidOperationException($"Job type name \"{registration.Name}\" is already registered for \"{existing.FullName}\"."); + + _nameToType[registration.Name] = registration.JobType; + _typeToName[registration.JobType] = registration.Name; + } +} + +public sealed class JobHandle +{ + private readonly IJobMonitor _monitor; + private readonly Func> _requestCancellation; + + internal JobHandle(string jobId, IJobMonitor monitor, Func> requestCancellation) + { + JobId = jobId; + _monitor = monitor; + _requestCancellation = requestCancellation; + } + + public string JobId { get; } + + public Task GetStateAsync(CancellationToken cancellationToken = default) + { + return _monitor.GetAsync(JobId, cancellationToken); + } + + public Task RequestCancellationAsync(CancellationToken cancellationToken = default) + { + return _requestCancellation(JobId, cancellationToken); + } +} + +/// +/// Passed to a job on each run. Gives the running job its identity and attempt number, plus store-backed progress +/// reporting, lease heartbeat (for long runs), and cooperative cancellation checks — the parts of +/// that are useful from inside job code. When a job is run outside the durable runtime +/// (for example directly in a test), the store-backed helpers are no-ops and cancellation reflects the supplied token. +/// +public sealed class JobExecutionContext +{ + private readonly IJobRuntimeStore? _store; + private readonly string _nodeId; + private readonly TimeSpan _lease; + private readonly ReadOnlyMemory? _payload; + private readonly string? _payloadType; + private readonly ISerializer? _serializer; + private readonly object? _detachedArguments; + + internal JobExecutionContext(string jobId, int attempt, CancellationToken cancellationToken, IJobRuntimeStore store, string nodeId, TimeSpan lease, ReadOnlyMemory? payload = null, string? payloadType = null, ISerializer? serializer = null) + { + JobId = jobId; + Attempt = attempt; + CancellationToken = cancellationToken; + _store = store; + _nodeId = nodeId; + _lease = lease; + _payload = payload; + _payloadType = payloadType; + _serializer = serializer; + } + + /// + /// Creates a detached context for running a job outside the durable runtime (tests or one-off invocations). + /// Progress reporting and lease renewal are no-ops; cancellation reflects ; + /// surfaces through without serialization. + /// + public JobExecutionContext(CancellationToken cancellationToken = default, string? jobId = null, int attempt = 1, object? arguments = null) + { + JobId = jobId ?? Guid.NewGuid().ToString("N"); + Attempt = attempt; + CancellationToken = cancellationToken; + _store = null; + _nodeId = String.Empty; + _lease = TimeSpan.Zero; + _detachedArguments = arguments; + } + + public string JobId { get; } + public int Attempt { get; } + public CancellationToken CancellationToken { get; } + + /// Whether this invocation carries typed arguments (see ). + public bool HasArguments => _detachedArguments is not null || _payload is not null; + + /// + /// The typed per-invocation arguments this job was enqueued with. Throws a descriptive + /// when the job was enqueued without arguments or the payload cannot be + /// read as (the stored discriminator is included for triage). + /// + public TArgs GetArguments() where TArgs : class + { + if (_detachedArguments is not null) + { + return _detachedArguments as TArgs + ?? throw new InvalidOperationException($"Job \"{JobId}\" arguments are of type \"{_detachedArguments.GetType().FullName}\", not the requested \"{typeof(TArgs).FullName}\"."); + } + + if (_payload is not { } payload) + throw new InvalidOperationException($"Job \"{JobId}\" was enqueued without arguments. Use EnqueueAsync(args) to supply a typed payload."); + + var serializer = _serializer ?? DefaultSerializer.Instance; + TArgs? args; + try + { + args = serializer.Deserialize(payload, typeof(TArgs)) as TArgs; + } + catch (Exception ex) + { + throw new InvalidOperationException($"Unable to deserialize job \"{JobId}\" arguments (stored type \"{_payloadType}\") as \"{typeof(TArgs).FullName}\".", ex); + } + + return args ?? throw new InvalidOperationException($"Job \"{JobId}\" arguments (stored type \"{_payloadType}\") deserialized to null as \"{typeof(TArgs).FullName}\"."); + } + + public Task ReportProgressAsync(int? percent = null, string? message = null, CancellationToken cancellationToken = default) + => _store?.SetProgressAsync(JobId, percent, message, cancellationToken) ?? Task.CompletedTask; + + // Extends the worker's lease so a long-but-alive run is not reclaimed as stale. + public Task RenewLeaseAsync(CancellationToken cancellationToken = default) + => _store?.RenewClaimAsync(JobId, _nodeId, _lease, cancellationToken) ?? Task.FromResult(true); + + public Task IsCancellationRequestedAsync(CancellationToken cancellationToken = default) + => _store?.IsCancellationRequestedAsync(JobId, cancellationToken) ?? Task.FromResult(CancellationToken.IsCancellationRequested); +} + +public interface IJobMonitor +{ + Task GetAsync(string jobId, CancellationToken cancellationToken = default); + Task> QueryAsync(JobQuery query, CancellationToken cancellationToken = default); +} + +public interface IJobClient +{ + Task EnqueueAsync(JobRequestOptions? options = null, CancellationToken cancellationToken = default) where TJob : IJob; + + /// + /// Enqueues a job with typed per-invocation arguments. The args are serialized into the durable + /// via the runtime's serializer and surface to the job through + /// . + /// + Task EnqueueAsync(TArgs args, JobRequestOptions? options = null, CancellationToken cancellationToken = default) where TJob : IJob where TArgs : class; + + Task EnqueueAsync(Type jobType, JobRequestOptions? options = null, CancellationToken cancellationToken = default); + Task RequestCancellationAsync(string jobId, CancellationToken cancellationToken = default); +} + +public interface IJobWorker +{ + Task RunAsync(string jobId, CancellationToken cancellationToken = default); + Task RunQueuedAsync(int limit = 100, CancellationToken cancellationToken = default); + // Reclaims jobs stuck in Processing past their lease (a worker that crashed mid-run): re-queues them while attempts + // remain, otherwise dead-letters them. Returns the number recovered. + Task RecoverStaleAsync(int maxAttempts, int limit = 100, CancellationToken cancellationToken = default); +} + +/// +/// Durable storage for time-gated dispatches: delayed messages beyond a transport's native ceiling, store-parked +/// retry delays, and CRON occurrence triggers. This is the only store contract the messaging client depends on — +/// a provider that offers durable scheduling without the full job runtime implements just this. +/// +public interface IScheduledDispatchStore +{ + Task ScheduleDispatchAsync(ScheduledDispatchState dispatch, CancellationToken cancellationToken = default); + Task> ClaimDueDispatchesAsync(DateTimeOffset now, int limit, string nodeId, TimeSpan lease, CancellationToken cancellationToken = default); + Task CompleteDispatchAsync(string dispatchId, string nodeId, CancellationToken cancellationToken = default); + Task ReleaseDispatchAsync(string dispatchId, string nodeId, DateTimeOffset nextDueUtc, CancellationToken cancellationToken = default); +} + +/// +/// The full job runtime store: job state persistence, queries, lease/ownership management, cancellation signaling, +/// and scheduled-dispatch storage. The state/lease/cancellation members are deliberately one contract — transitions +/// verify ownership atomically (see / ), so +/// splitting them would break the compare-and-set semantics correctness depends on. +/// +public interface IJobRuntimeStore : IJobMonitor, IScheduledDispatchStore +{ + Task CreateIfAbsentAsync(JobState initial, CancellationToken cancellationToken = default); + // When expectedNodeId is non-null, the transition only succeeds if the job is currently owned by that node. + // Worker terminal transitions pass their node id so a stale worker whose lease was reclaimed cannot overwrite + // the new owner's state. + Task TryTransitionAsync(string jobId, JobStatus expectedStatus, JobStatus newStatus, JobStatePatch? patch = null, string? expectedNodeId = null, CancellationToken cancellationToken = default); + Task TryClaimAsync(string jobId, string nodeId, TimeSpan lease, CancellationToken cancellationToken = default); + Task RenewClaimAsync(string jobId, string nodeId, TimeSpan lease, CancellationToken cancellationToken = default); + Task ReleaseClaimAsync(string jobId, string nodeId, CancellationToken cancellationToken = default); + // Returns plain (non-CRON-occurrence) jobs in Processing whose lease has expired as of + // (their owning worker is presumed dead), so the runtime can reclaim them. CRON occurrences are excluded — the + // scheduler recovers those with its own per-definition retry budget. + Task> GetExpiredProcessingAsync(DateTimeOffset now, int limit, CancellationToken cancellationToken = default); + // Atomically reclaims a stale Processing job: the transition applies only if the job is STILL owned by + // and its lease is STILL expired as of . This closes the + // race where the owning worker renews its lease between a stale scan and the reclaim (which would otherwise + // re-queue a live job and double-run it). + Task TryReclaimExpiredAsync(string jobId, DateTimeOffset now, string expectedNodeId, JobStatus newStatus, JobStatePatch? patch = null, CancellationToken cancellationToken = default); + Task SetProgressAsync(string jobId, int? percent = null, string? message = null, CancellationToken cancellationToken = default); + Task IncrementAttemptAsync(string jobId, CancellationToken cancellationToken = default); + Task RequestCancellationAsync(string jobId, CancellationToken cancellationToken = default); + Task IsCancellationRequestedAsync(string jobId, CancellationToken cancellationToken = default); +} + +public sealed class InMemoryJobRuntimeStore : IJobRuntimeStore +{ + private readonly ConcurrentDictionary _jobs = new(StringComparer.Ordinal); + private readonly ConcurrentDictionary _dispatches = new(StringComparer.Ordinal); + private readonly TimeProvider _timeProvider; + private readonly object _lock = new(); + + public InMemoryJobRuntimeStore(TimeProvider? timeProvider = null) + { + _timeProvider = timeProvider ?? TimeProvider.System; + } + + public Task CreateIfAbsentAsync(JobState initial, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(initial); + cancellationToken.ThrowIfCancellationRequested(); + + var now = _timeProvider.GetUtcNow(); + _jobs.TryAdd(initial.JobId, initial with + { + CreatedUtc = initial.CreatedUtc == default ? now : initial.CreatedUtc, + LastUpdatedUtc = initial.LastUpdatedUtc == default ? now : initial.LastUpdatedUtc + }); + + return Task.CompletedTask; + } + + public Task GetAsync(string jobId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + _jobs.TryGetValue(jobId, out var state); + return Task.FromResult(state); + } + + public Task> QueryAsync(JobQuery query, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(query); + cancellationToken.ThrowIfCancellationRequested(); + + IEnumerable results = _jobs.Values; + if (!String.IsNullOrEmpty(query.Name)) + results = results.Where(s => String.Equals(s.Name, query.Name, StringComparison.Ordinal)); + + if (query.Status is { } status) + results = results.Where(s => s.Status == status); + + if (query.ExcludeOccurrences) + results = results.Where(s => s.ScheduledForUtc is null); + + return Task.FromResult>(results + .OrderByDescending(s => s.LastUpdatedUtc) + .Take(Math.Max(1, query.Limit)) + .ToArray()); + } + + public Task TryTransitionAsync(string jobId, JobStatus expectedStatus, JobStatus newStatus, JobStatePatch? patch = null, string? expectedNodeId = null, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + lock (_lock) + { + if (!_jobs.TryGetValue(jobId, out var current) || current.Status != expectedStatus) + return Task.FromResult(false); + + if (expectedNodeId is not null && !String.Equals(current.NodeId, expectedNodeId, StringComparison.Ordinal)) + return Task.FromResult(false); + + _jobs[jobId] = ApplyPatch(current, patch) with + { + Status = newStatus, + LastUpdatedUtc = patch?.LastUpdatedUtc ?? _timeProvider.GetUtcNow() + }; + return Task.FromResult(true); + } + } + + public Task TryClaimAsync(string jobId, string nodeId, TimeSpan lease, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + ArgumentException.ThrowIfNullOrEmpty(nodeId); + + lock (_lock) + { + if (!_jobs.TryGetValue(jobId, out var current)) + return Task.FromResult(false); + + var now = _timeProvider.GetUtcNow(); + if (!String.IsNullOrEmpty(current.NodeId) && current.LeaseExpiresUtc is { } leaseExpires && leaseExpires > now && current.NodeId != nodeId) + return Task.FromResult(false); + + _jobs[jobId] = current with + { + NodeId = nodeId, + LeaseExpiresUtc = now.Add(lease), + LastUpdatedUtc = now + }; + return Task.FromResult(true); + } + } + + public Task RenewClaimAsync(string jobId, string nodeId, TimeSpan lease, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + lock (_lock) + { + if (!_jobs.TryGetValue(jobId, out var current) || current.NodeId != nodeId) + return Task.FromResult(false); + + var now = _timeProvider.GetUtcNow(); + _jobs[jobId] = current with + { + LeaseExpiresUtc = now.Add(lease), + LastUpdatedUtc = now + }; + return Task.FromResult(true); + } + } + + public Task ReleaseClaimAsync(string jobId, string nodeId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + lock (_lock) + { + if (!_jobs.TryGetValue(jobId, out var current) || current.NodeId != nodeId) + return Task.FromResult(false); + + _jobs[jobId] = current with + { + NodeId = null, + LeaseExpiresUtc = null, + LastUpdatedUtc = _timeProvider.GetUtcNow() + }; + return Task.FromResult(true); + } + } + + public Task> GetExpiredProcessingAsync(DateTimeOffset now, int limit, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + lock (_lock) + { + var expired = _jobs.Values + // Exclude CRON occurrences (ScheduledForUtc set): the scheduler owns their recovery via its own + // per-definition retry budget. This path only recovers plain IJobClient-submitted jobs. + .Where(s => s.Status == JobStatus.Processing && s.ScheduledForUtc is null && s.LeaseExpiresUtc is { } lease && lease <= now) + .OrderBy(s => s.LeaseExpiresUtc) + .Take(Math.Max(1, limit)) + .ToArray(); + + return Task.FromResult>(expired); + } + } + + public Task TryReclaimExpiredAsync(string jobId, DateTimeOffset now, string expectedNodeId, JobStatus newStatus, JobStatePatch? patch = null, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + ArgumentException.ThrowIfNullOrEmpty(expectedNodeId); + + lock (_lock) + { + if (!_jobs.TryGetValue(jobId, out var current)) + return Task.FromResult(false); + + // Re-check (atomically, under the lock) the conditions the stale scan saw: still Processing, still owned by + // the same node, and the lease is still expired. A renewal or re-claim that landed since the scan fails one + // of these and the reclaim is skipped. + if (current.Status != JobStatus.Processing || !String.Equals(current.NodeId, expectedNodeId, StringComparison.Ordinal)) + return Task.FromResult(false); + + if (current.LeaseExpiresUtc is not { } lease || lease > now) + return Task.FromResult(false); + + _jobs[jobId] = ApplyPatch(current, patch) with + { + Status = newStatus, + LastUpdatedUtc = patch?.LastUpdatedUtc ?? _timeProvider.GetUtcNow() + }; + return Task.FromResult(true); + } + } + + public Task SetProgressAsync(string jobId, int? percent = null, string? message = null, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + UpdateJob(jobId, state => state with + { + Progress = percent ?? state.Progress, + ProgressMessage = message ?? state.ProgressMessage, + LastUpdatedUtc = _timeProvider.GetUtcNow() + }); + + return Task.CompletedTask; + } + + public Task IncrementAttemptAsync(string jobId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + UpdateJob(jobId, state => state with { Attempt = state.Attempt + 1, LastUpdatedUtc = _timeProvider.GetUtcNow() }); + return Task.CompletedTask; + } + + public Task RequestCancellationAsync(string jobId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + return Task.FromResult(UpdateJob(jobId, state => state with + { + CancellationRequested = true, + LastUpdatedUtc = _timeProvider.GetUtcNow() + })); + } + + public Task IsCancellationRequestedAsync(string jobId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + return Task.FromResult(_jobs.TryGetValue(jobId, out var state) && state.CancellationRequested); + } + + public Task ScheduleDispatchAsync(ScheduledDispatchState dispatch, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(dispatch); + cancellationToken.ThrowIfCancellationRequested(); + _dispatches.TryAdd(dispatch.DispatchId, dispatch); + return Task.CompletedTask; + } + + public Task> ClaimDueDispatchesAsync(DateTimeOffset now, int limit, string nodeId, TimeSpan lease, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + ArgumentException.ThrowIfNullOrEmpty(nodeId); + + lock (_lock) + { + var due = _dispatches.Values + .Where(d => d.DueUtc <= now && (String.IsNullOrEmpty(d.ClaimOwner) || d.ClaimExpiresUtc <= now)) + .OrderBy(d => d.DueUtc) + .Take(Math.Max(1, limit)) + .ToArray(); + + for (int index = 0; index < due.Length; index++) + { + var claimed = due[index] with + { + ClaimOwner = nodeId, + ClaimExpiresUtc = now.Add(lease), + Attempts = due[index].Attempts + 1 + }; + _dispatches[claimed.DispatchId] = claimed; + due[index] = claimed; + } + + return Task.FromResult>(due); + } + } + + public Task CompleteDispatchAsync(string dispatchId, string nodeId, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + lock (_lock) + { + if (_dispatches.TryGetValue(dispatchId, out var dispatch) && dispatch.ClaimOwner == nodeId) + _dispatches.TryRemove(dispatchId, out _); + } + + return Task.CompletedTask; + } + + public Task ReleaseDispatchAsync(string dispatchId, string nodeId, DateTimeOffset nextDueUtc, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + lock (_lock) + { + if (_dispatches.TryGetValue(dispatchId, out var dispatch) && dispatch.ClaimOwner == nodeId) + { + _dispatches[dispatchId] = dispatch with + { + DueUtc = nextDueUtc, + ClaimOwner = null, + ClaimExpiresUtc = null + }; + } + } + + return Task.CompletedTask; + } + + private bool UpdateJob(string jobId, Func update) + { + lock (_lock) + { + if (!_jobs.TryGetValue(jobId, out var current)) + return false; + + _jobs[jobId] = update(current); + return true; + } + } + + private JobState ApplyPatch(JobState state, JobStatePatch? patch) + { + if (patch is null) + return state; + + return state with + { + Status = patch.Status ?? state.Status, + JobType = patch.JobType ?? state.JobType, + Progress = patch.Progress ?? state.Progress, + ProgressMessage = patch.ProgressMessage ?? state.ProgressMessage, + Error = patch.Error ?? state.Error, + Attempt = state.Attempt + patch.AttemptDelta, + NodeId = patch.ClearNodeId ? null : patch.NodeId ?? state.NodeId, + LeaseExpiresUtc = patch.ClearLeaseExpiresUtc ? null : patch.LeaseExpiresUtc ?? state.LeaseExpiresUtc, + LastUpdatedUtc = patch.LastUpdatedUtc ?? state.LastUpdatedUtc, + StartedUtc = patch.StartedUtc ?? state.StartedUtc, + CompletedUtc = patch.CompletedUtc ?? state.CompletedUtc, + CancellationRequested = patch.CancellationRequested ?? state.CancellationRequested + }; + } +} + +public sealed class JobClient : IJobClient +{ + private readonly IJobRuntimeStore _store; + private readonly TimeProvider _timeProvider; + private readonly IJobTypeRegistry _jobTypes; + private readonly ISerializer _serializer; + + public JobClient(IJobRuntimeStore store, TimeProvider? timeProvider = null, IJobTypeRegistry? jobTypes = null, ISerializer? serializer = null) + { + _store = store ?? throw new ArgumentNullException(nameof(store)); + _timeProvider = timeProvider ?? TimeProvider.System; + _jobTypes = jobTypes ?? new JobTypeRegistry(); + _serializer = serializer ?? DefaultSerializer.Instance; + } + + public Task EnqueueAsync(JobRequestOptions? options = null, CancellationToken cancellationToken = default) where TJob : IJob + { + return EnqueueCoreAsync(typeof(TJob), args: null, options, cancellationToken); + } + + public Task EnqueueAsync(TArgs args, JobRequestOptions? options = null, CancellationToken cancellationToken = default) where TJob : IJob where TArgs : class + { + ArgumentNullException.ThrowIfNull(args); + return EnqueueCoreAsync(typeof(TJob), args, options, cancellationToken); + } + + public Task EnqueueAsync(Type jobType, JobRequestOptions? options = null, CancellationToken cancellationToken = default) + { + return EnqueueCoreAsync(jobType, args: null, options, cancellationToken); + } + + private async Task EnqueueCoreAsync(Type jobType, object? args, JobRequestOptions? options, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(jobType); + if (!typeof(IJob).IsAssignableFrom(jobType)) + throw new ArgumentException("Job type must implement IJob.", nameof(jobType)); + + options ??= new JobRequestOptions(); + string jobId = options.JobId ?? Guid.NewGuid().ToString("N"); + string name = options.Name ?? jobType.Name; + var now = _timeProvider.GetUtcNow(); + + await _store.CreateIfAbsentAsync(new JobState + { + JobId = jobId, + Name = name, + JobType = _jobTypes.GetName(jobType), + // Explicitly typed: the byte[] -> ReadOnlyMemory conversion maps a null array to an EMPTY memory, which + // would make an argless job look like it carries a zero-byte payload. + Payload = args is null ? null : (ReadOnlyMemory?)_serializer.SerializeToBytes(args), + PayloadType = args?.GetType().FullName, + Status = JobStatus.Queued, + CreatedUtc = now, + LastUpdatedUtc = now + }, cancellationToken).ConfigureAwait(false); + + return new JobHandle(jobId, _store, RequestCancellationAsync); + } + + public Task RequestCancellationAsync(string jobId, CancellationToken cancellationToken = default) + { + return _store.RequestCancellationAsync(jobId, cancellationToken); + } +} + +/// +/// Resolves a stable, process-unique node identity used for job claims and per-node scheduling. +/// Honors the FOUNDATIO_NODE_ID environment variable when set; otherwise combines machine name, +/// process id, and a process-lifetime token so co-located worker processes do not collapse to one identity. +/// +internal static class NodeIdentity +{ + public static string Current { get; } = Resolve(); + + private static string Resolve() + { + string? configured = Environment.GetEnvironmentVariable("FOUNDATIO_NODE_ID"); + if (!String.IsNullOrEmpty(configured)) + return configured; + + return $"{Environment.MachineName}:{Environment.ProcessId}:{Guid.NewGuid().ToString("N")[..8]}"; + } +} + +public sealed class JobWorker : IJobWorker +{ + private static readonly TimeSpan DefaultLease = TimeSpan.FromMinutes(5); + private static readonly TimeSpan DefaultCancellationPollInterval = TimeSpan.FromSeconds(1); + + private readonly IJobRuntimeStore _store; + private readonly IServiceProvider _serviceProvider; + private readonly TimeProvider _timeProvider; + private readonly IJobTypeRegistry _jobTypes; + private readonly ISerializer _serializer; + private readonly string _nodeId; + private readonly TimeSpan _lease; + private readonly TimeSpan _cancellationPollInterval; + private readonly int _maxConcurrency; + + public JobWorker(IJobRuntimeStore store, IServiceProvider serviceProvider, TimeProvider? timeProvider = null, string? nodeId = null, TimeSpan? lease = null, IJobTypeRegistry? jobTypes = null, TimeSpan? cancellationPollInterval = null, ISerializer? serializer = null, int maxConcurrency = 1) + { + _store = store ?? throw new ArgumentNullException(nameof(store)); + _serviceProvider = serviceProvider ?? throw new ArgumentNullException(nameof(serviceProvider)); + _timeProvider = timeProvider ?? TimeProvider.System; + _jobTypes = jobTypes ?? new JobTypeRegistry(); + _serializer = serializer ?? DefaultSerializer.Instance; + _nodeId = !String.IsNullOrEmpty(nodeId) ? nodeId : NodeIdentity.Current; + _lease = lease ?? DefaultLease; + + // Default 1 preserves per-node ordering and today's behavior; raise for I/O-bound jobs. Each in-flight job + // still gets its own DI scope, lease renewal, and cancellation watcher. + _maxConcurrency = Math.Max(1, maxConcurrency); + + // Cooperative cancellation is observed by polling the runtime store. The default is intentionally + // conservative (one poll per second per running job) so a real store isn't hammered when many jobs run + // concurrently; callers that need snappier cancellation can opt into a tighter interval. + var pollInterval = cancellationPollInterval ?? DefaultCancellationPollInterval; + _cancellationPollInterval = pollInterval > TimeSpan.Zero ? pollInterval : DefaultCancellationPollInterval; + } + + public async Task RunQueuedAsync(int limit = 100, CancellationToken cancellationToken = default) + { + var queued = await _store.QueryAsync(new JobQuery + { + Status = JobStatus.Queued, + Limit = limit, + // The scheduler owns CRON occurrences (retry/dead-letter accounting); the generic worker must skip them. + ExcludeOccurrences = true + }, cancellationToken).ConfigureAwait(false); + + if (_maxConcurrency <= 1) + { + int sequentialCompleted = 0; + foreach (var state in queued) + { + if (await RunJobStateAsync(state, cancellationToken).ConfigureAwait(false)) + sequentialCompleted++; + } + + return sequentialCompleted; + } + + // Bounded pool: at most _maxConcurrency jobs in flight; a slot frees the moment a job settles, so one slow + // job never idles the rest of the batch. Claims are TryTransition-guarded, so concurrency cannot double-run. + int completed = 0; + using var slots = new SemaphoreSlim(_maxConcurrency, _maxConcurrency); + var inFlight = new List(queued.Count); + + foreach (var state in queued) + { + await slots.WaitAsync(cancellationToken).ConfigureAwait(false); + inFlight.Add(Task.Run(async () => + { + try + { + if (await RunJobStateAsync(state, cancellationToken).ConfigureAwait(false)) + Interlocked.Increment(ref completed); + } + finally + { + slots.Release(); + } + }, CancellationToken.None)); + } + + await Task.WhenAll(inFlight).ConfigureAwait(false); + return completed; + } + + public async Task RunAsync(string jobId, CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrEmpty(jobId); + + var state = await _store.GetAsync(jobId, cancellationToken).ConfigureAwait(false); + return state is not null && await RunJobStateAsync(state, cancellationToken).ConfigureAwait(false); + } + + public async Task RecoverStaleAsync(int maxAttempts, int limit = 100, CancellationToken cancellationToken = default) + { + var now = _timeProvider.GetUtcNow(); + var stale = await _store.GetExpiredProcessingAsync(now, limit, cancellationToken).ConfigureAwait(false); + + int recovered = 0; + foreach (var state in stale) + { + if (String.IsNullOrEmpty(state.NodeId)) + continue; + + // TryReclaimExpiredAsync re-verifies (atomically) that the job is still owned by the same presumed-dead + // node and its lease is still expired, so a worker that renewed between the scan and here is not yanked out + // from under itself (no double-run). Attempts are incremented per run, so a job that keeps crashing is + // dead-lettered once it has consumed its attempt budget instead of being re-queued forever. + // + // Budget semantics for ad-hoc (IJobClient) jobs: `maxAttempts` is the TOTAL number of attempts, so + // dead-letter at Attempt >= maxAttempts. (CRON occurrences use a different knob — ScheduledJobDefinition + // .MaxRetries, the number of retries AFTER the first run, i.e. total runs = MaxRetries + 1 — and are + // excluded from this path via GetExpiredProcessingAsync; the scheduler owns their recovery.) + bool transitioned = state.Attempt >= maxAttempts + ? await _store.TryReclaimExpiredAsync(state.JobId, now, state.NodeId, JobStatus.DeadLettered, new JobStatePatch + { + Error = $"Lease expired after {state.Attempt} attempt(s) without completion.", + ClearNodeId = true, + ClearLeaseExpiresUtc = true, + CompletedUtc = now, + LastUpdatedUtc = now + }, cancellationToken).ConfigureAwait(false) + : await _store.TryReclaimExpiredAsync(state.JobId, now, state.NodeId, JobStatus.Queued, new JobStatePatch + { + ClearNodeId = true, + ClearLeaseExpiresUtc = true, + LastUpdatedUtc = now + }, cancellationToken).ConfigureAwait(false); + + if (transitioned) + recovered++; + } + + return recovered; + } + + private async Task RunJobStateAsync(JobState state, CancellationToken cancellationToken) + { + if (state.Status != JobStatus.Queued) + return false; + + var now = _timeProvider.GetUtcNow(); + if (!await _store.TryTransitionAsync(state.JobId, JobStatus.Queued, JobStatus.Processing, new JobStatePatch + { + NodeId = _nodeId, + StartedUtc = now, + LeaseExpiresUtc = now.Add(_lease), + AttemptDelta = 1 + }, cancellationToken: cancellationToken).ConfigureAwait(false)) + { + return false; + } + + var jobTag = new KeyValuePair("job", state.Name); + JobInstruments.Started.Add(1, jobTag); + + using var linkedCancellationTokenSource = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + + // Lease renewal and cancellation polling are supervised loops owned by this run — started here, stopped and + // awaited when the run ends — not fire-and-forget timers whose failures would vanish unobserved. + using var supervisionCancellationTokenSource = new CancellationTokenSource(); + var leaseLoop = Task.Run(() => RunLeaseRenewalLoopAsync(state.JobId, linkedCancellationTokenSource, supervisionCancellationTokenSource.Token), CancellationToken.None); + var cancellationLoop = Task.Run(() => RunCancellationPollLoopAsync(state.JobId, linkedCancellationTokenSource, supervisionCancellationTokenSource.Token), CancellationToken.None); + + try + { + var jobType = ResolveJobType(state); + + // Hand the job its execution context (identity, attempt, typed payload, progress, heartbeat, cancellation). + // The store was already incremented to this attempt by the Queued -> Processing transition above. + var context = new JobExecutionContext(state.JobId, state.Attempt + 1, linkedCancellationTokenSource.Token, _store, _nodeId, _lease, state.Payload, state.PayloadType, _serializer); + + var result = await ExecuteJobAsync(jobType, context).ConfigureAwait(false); + var completedAt = _timeProvider.GetUtcNow(); + + if (result.IsCancelled) + { + await _store.TryTransitionAsync(state.JobId, JobStatus.Processing, JobStatus.Cancelled, new JobStatePatch + { + Error = result.Message, + CompletedUtc = completedAt, + ClearNodeId = true, + ClearLeaseExpiresUtc = true + }, expectedNodeId: _nodeId, cancellationToken: CancellationToken.None).ConfigureAwait(false); + } + else if (result.IsSuccess) + { + await _store.TryTransitionAsync(state.JobId, JobStatus.Processing, JobStatus.Completed, new JobStatePatch + { + CompletedUtc = completedAt, + ClearNodeId = true, + ClearLeaseExpiresUtc = true, + Progress = 100 + }, expectedNodeId: _nodeId, cancellationToken: CancellationToken.None).ConfigureAwait(false); + } + else + { + await _store.TryTransitionAsync(state.JobId, JobStatus.Processing, JobStatus.Failed, new JobStatePatch + { + Error = result.Message, + CompletedUtc = completedAt, + ClearNodeId = true, + ClearLeaseExpiresUtc = true + }, expectedNodeId: _nodeId, cancellationToken: CancellationToken.None).ConfigureAwait(false); + } + + if (result.IsCancelled) + JobInstruments.Cancelled.Add(1, jobTag); + else if (result.IsSuccess) + JobInstruments.Completed.Add(1, jobTag); + else + JobInstruments.Failed.Add(1, jobTag); + + JobInstruments.RunTime.Record((completedAt - now).TotalMilliseconds, jobTag); + return true; + } + catch (Exception ex) + { + var failedAt = _timeProvider.GetUtcNow(); + await _store.TryTransitionAsync(state.JobId, JobStatus.Processing, JobStatus.Failed, new JobStatePatch + { + Error = ex.Message, + CompletedUtc = failedAt, + ClearNodeId = true, + ClearLeaseExpiresUtc = true + }, expectedNodeId: _nodeId, cancellationToken: CancellationToken.None).ConfigureAwait(false); + JobInstruments.Failed.Add(1, jobTag); + JobInstruments.RunTime.Record((failedAt - now).TotalMilliseconds, jobTag); + throw; + } + finally + { + // Stop and await the supervision loops so no renewal/poll outlives its run (and so their final state is + // observed rather than dropped on the floor). The loops never throw; they classify failures themselves. + await supervisionCancellationTokenSource.CancelAsync().ConfigureAwait(false); + await Task.WhenAll(leaseLoop, cancellationLoop).ConfigureAwait(false); + } + } + + // Every execution gets its own async DI scope, owned for exactly the run: scoped services (DbContexts, units of + // work) resolve per run and are disposed when it ends, instead of silently resolving as effective singletons from + // the root container. A bare provider without scope support (custom IServiceProvider) runs unscoped. + private async Task ExecuteJobAsync(Type jobType, JobExecutionContext context) + { + if (_serviceProvider.GetService(typeof(IServiceScopeFactory)) is IServiceScopeFactory scopeFactory) + { + await using var scope = scopeFactory.CreateAsyncScope(); + var job = (IJob)ActivatorUtilities.GetServiceOrCreateInstance(scope.ServiceProvider, jobType); + return await job.TryRunAsync(context).ConfigureAwait(false); + } + + var unscoped = (IJob)ActivatorUtilities.GetServiceOrCreateInstance(_serviceProvider, jobType); + return await unscoped.TryRunAsync(context).ConfigureAwait(false); + } + + private Type ResolveJobType(JobState state) + { + if (String.IsNullOrEmpty(state.JobType)) + throw new InvalidOperationException($"Job \"{state.JobId}\" does not have a job type and cannot be executed by a worker."); + + try + { + return _jobTypes.Resolve(state.JobType); + } + catch (Exception ex) when (ex is InvalidOperationException or ArgumentException) + { + throw new InvalidOperationException($"Job type \"{state.JobType}\" for job \"{state.JobId}\" could not be resolved to an IJob implementation.", ex); + } + } + + // Lease renewal supervises its own failures. A clean "renewal denied" means the lease was lost to another node. + // Renewal that keeps THROWING (a store outage) is treated the same once the lease window passes without one + // success — the lease has lapsed on the broker's clock too, so another node may already have reclaimed the job, + // and letting this run continue would double-execute its side effects. Both paths cancel the run; the terminal + // transition (guarded by expectedNodeId) then cannot overwrite the new owner's state. + private async Task RunLeaseRenewalLoopAsync(string jobId, CancellationTokenSource jobCancellation, CancellationToken supervision) + { + // Renew well before the lease elapses so a slow-but-alive worker keeps ownership and is not reclaimed. + var interval = TimeSpan.FromMilliseconds(Math.Max(250, _lease.TotalMilliseconds / 3)); + long lastSuccessTimestamp = _timeProvider.GetTimestamp(); + + while (!supervision.IsCancellationRequested) + { + await _timeProvider.SafeDelay(interval, supervision).ConfigureAwait(false); + if (supervision.IsCancellationRequested) + return; + + try + { + if (!await _store.RenewClaimAsync(jobId, _nodeId, _lease, CancellationToken.None).ConfigureAwait(false)) + { + await CancelRunAsync(jobCancellation).ConfigureAwait(false); + return; + } + + lastSuccessTimestamp = _timeProvider.GetTimestamp(); + } + catch (Exception) + { + // Transient store failure: retry next tick — but never outlive the lease on hope. + if (_timeProvider.GetElapsedTime(lastSuccessTimestamp) >= _lease) + { + await CancelRunAsync(jobCancellation).ConfigureAwait(false); + return; + } + } + } + } + + // Cancellation polling keeps polling through store failures (a missed poll only delays cooperative cancellation, + // it cannot double-run anything), and stops when the run ends or cancellation is observed. + private async Task RunCancellationPollLoopAsync(string jobId, CancellationTokenSource jobCancellation, CancellationToken supervision) + { + while (!supervision.IsCancellationRequested) + { + await _timeProvider.SafeDelay(_cancellationPollInterval, supervision).ConfigureAwait(false); + if (supervision.IsCancellationRequested) + return; + + try + { + if (await _store.IsCancellationRequestedAsync(jobId, CancellationToken.None).ConfigureAwait(false)) + { + await CancelRunAsync(jobCancellation).ConfigureAwait(false); + return; + } + } + catch (Exception) + { + } + } + } + + private static async Task CancelRunAsync(CancellationTokenSource jobCancellation) + { + try + { + await jobCancellation.CancelAsync().ConfigureAwait(false); + } + catch (ObjectDisposedException) + { + // The run already completed and disposed its token source; nothing left to cancel. + } + } +} diff --git a/src/Foundatio/Jobs/JobRuntimePumpService.cs b/src/Foundatio/Jobs/JobRuntimePumpService.cs new file mode 100644 index 000000000..fb4a01502 --- /dev/null +++ b/src/Foundatio/Jobs/JobRuntimePumpService.cs @@ -0,0 +1,166 @@ +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Utility; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; + +namespace Foundatio.Jobs; + +/// Cadence and batch size for the durable job-runtime pump. +public class JobRuntimePumpOptions +{ + /// + /// Whether the auto-registered runtime pump runs. Default true. Set false to take manual control of pumping (e.g. + /// drive / yourself, or run the pump on only some nodes); + /// the hosted service is then registered but does nothing. Configure via AddFoundatio().Jobs.ConfigureRuntimePump + /// or AddJobRuntimeService. + /// + public bool Enabled { get; set; } = true; + + /// How often the pump materializes CRON occurrences, dispatches due work, and runs queued jobs. Default 1s. + public TimeSpan PollInterval { get; set; } = TimeSpan.FromSeconds(1); + + /// Maximum number of due dispatches and queued jobs claimed per iteration. Default 100. + public int BatchSize { get; set; } = 100; + + /// Maximum processing attempts for an ad-hoc job before a stale (lease-expired) instance is dead-lettered. Default 3. + public int MaxJobAttempts { get; set; } = 3; + + /// + /// Maximum queued jobs the worker executes concurrently per node. Default 1, which preserves per-node run + /// ordering; raise it for I/O-bound jobs. Every in-flight job gets its own DI scope, lease, and cancellation watcher. + /// + public int WorkerConcurrency { get; set; } = 1; +} + +/// +/// Drives the durable job runtime (): materializes CRON occurrences, dispatches +/// delayed/scheduled work (including the messaging delayed-delivery fallback), recovers stale occurrences, and runs +/// jobs submitted via . Registered automatically whenever a runtime store is configured +/// (AddFoundatio().Jobs.UseInMemoryRuntime() / UseRuntimeStore()) so a configured store can never +/// silently accumulate work that nothing drains. In a non-hosted process (no generic host) it is simply never started. +/// +public class JobRuntimePumpService : BackgroundService +{ + private readonly JobScheduleProcessor _processor; + private readonly IJobWorker _worker; + private readonly TimeProvider _timeProvider; + private readonly ILogger _logger; + private readonly JobRuntimePumpOptions _options; + private readonly IJobScheduler? _scheduler; + private readonly IEnumerable _scheduledJobs; + + public JobRuntimePumpService(JobScheduleProcessor processor, IJobWorker worker, TimeProvider? timeProvider = null, ILoggerFactory? loggerFactory = null, JobRuntimePumpOptions? options = null, IJobScheduler? scheduler = null, IEnumerable? scheduledJobs = null) + { + _processor = processor ?? throw new ArgumentNullException(nameof(processor)); + _worker = worker ?? throw new ArgumentNullException(nameof(worker)); + _timeProvider = timeProvider ?? TimeProvider.System; + _logger = (loggerFactory ?? NullLoggerFactory.Instance).CreateLogger(); + _options = options ?? new JobRuntimePumpOptions(); + _scheduler = scheduler; + _scheduledJobs = scheduledJobs ?? Array.Empty(); + } + + protected override async Task ExecuteAsync(CancellationToken stoppingToken) + { + // Schedule CRON jobs registered declaratively via AddFoundatio().Jobs.AddCronJob() so users don't have to + // call IJobScheduler.ScheduleAsync themselves. Done before the Enabled check so the "scheduled automatically" + // contract holds even when this node's pump is disabled for manual control. Idempotent (schedule keyed by name), + // so every node registering the same schedules is fine. + if (_scheduler is not null) + { + foreach (var definition in _scheduledJobs) + { + try + { + await _scheduler.ScheduleAsync(definition, stoppingToken).AnyContext(); + } + catch (Exception ex) + { + _logger.LogError(ex, "Failed to schedule CRON job {JobName}: {Message}", definition.Name, ex.Message); + } + } + } + + if (!_options.Enabled) + { + _logger.LogInformation("Job runtime pump disabled (JobRuntimePumpOptions.Enabled = false); not pumping the runtime store"); + return; + } + + _logger.LogInformation("Job runtime pump starting (poll interval {PollInterval}, batch size {BatchSize}, worker concurrency {WorkerConcurrency})", _options.PollInterval, _options.BatchSize, _options.WorkerConcurrency); + + // Execution (dispatching due work and running jobs) is an overlapped pass: the scheduling stage below keeps + // its cadence every poll even while a long job runs, so CRON materialization and the messaging delayed- + // delivery fallback are never head-of-line blocked by job duration. At most one pass is in flight; if the + // prior pass is still running when the loop comes around, this tick only materializes. Overlap-adjacent races + // are safe: dispatch claims are leased and job claims are TryTransition-guarded, so nothing double-runs. + var executionPass = Task.CompletedTask; + + while (!stoppingToken.IsCancellationRequested) + { + try + { + var now = _timeProvider.GetUtcNow(); + + // Materialize CRON occurrences due within the misfire window (deduped, idempotent). + await _processor.EnqueueDueOccurrencesAsync(now, stoppingToken).AnyContext(); + + if (executionPass.IsCompleted) + executionPass = Task.Run(() => RunExecutionPassAsync(now, stoppingToken), CancellationToken.None); + } + catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) + { + break; + } + catch (Exception ex) + { + _logger.LogError(ex, "Error pumping job runtime: {Message}", ex.Message); + } + + try + { + await _timeProvider.Delay(_options.PollInterval, stoppingToken).AnyContext(); + } + catch (OperationCanceledException) + { + break; + } + } + + // Drain the in-flight pass so shutdown does not abandon running jobs mid-settlement. + try + { + await executionPass.AnyContext(); + } + catch (OperationCanceledException) { } + + _logger.LogInformation("Job runtime pump stopped"); + } + + private async Task RunExecutionPassAsync(DateTimeOffset now, CancellationToken stoppingToken) + { + try + { + // Claim and run due dispatches: delayed queue/pub-sub messages first, then CRON occurrences, recovering + // occurrences whose processing lease expired and applying retry/dead-letter. + await _processor.RunDueOccurrencesAsync(now, _options.BatchSize, lease: null, stoppingToken).AnyContext(); + + // Recover ad-hoc (non-CRON) jobs whose processing lease expired (a worker crash mid-run). + await _worker.RecoverStaleAsync(_options.MaxJobAttempts, _options.BatchSize, stoppingToken).AnyContext(); + + // Run jobs submitted via IJobClient sitting in the Queued state. + await _worker.RunQueuedAsync(_options.BatchSize, stoppingToken).AnyContext(); + } + catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested) + { + } + catch (Exception ex) + { + _logger.LogError(ex, "Error running job runtime execution pass: {Message}", ex.Message); + } + } +} diff --git a/src/Foundatio/Jobs/JobScheduler.cs b/src/Foundatio/Jobs/JobScheduler.cs new file mode 100644 index 000000000..623f3c3b9 --- /dev/null +++ b/src/Foundatio/Jobs/JobScheduler.cs @@ -0,0 +1,460 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Cronos; +using Foundatio.Serializer; +using Foundatio.Messaging; + +namespace Foundatio.Jobs; + +public enum ScheduledJobScope +{ + Global, + PerNode +} + +public enum OverlapPolicy +{ + SkipIfRunning, + AllowConcurrent +} + +public sealed record ScheduledJobDefinition +{ + public required string Name { get; init; } + public required string Cron { get; init; } + public Type? JobType { get; init; } + public TimeZoneInfo? TimeZone { get; init; } + public ScheduledJobScope Scope { get; init; } = ScheduledJobScope.Global; + public OverlapPolicy Overlap { get; init; } = OverlapPolicy.SkipIfRunning; + public TimeSpan? MisfireWindow { get; init; } + public int MaxRetries { get; init; } = 3; + + /// + /// Computes the delay before a failed occurrence is retried, given the attempt number (1-based). + /// Defaults to capped exponential backoff when null. + /// + public Func? RetryBackoff { get; init; } + + /// + /// Typed arguments serialized into every occurrence's ; the job reads them via + /// . Null when the job takes none. + /// + public object? Arguments { get; init; } + + public bool Enabled { get; init; } = true; +} + +/// +/// Options for a declaratively-registered CRON job — AddFoundatio().Jobs.AddCronJob<TJob>(cron, o => ...). +/// The registered definitions are scheduled automatically when the runtime pump starts. +/// +public sealed class CronJobOptions +{ + /// Schedule name (must be unique across scheduled jobs). Defaults to the job type name. + public string? Name { get; set; } + + /// Global (one instance per tick, the default) or PerNode (every instance runs it per tick). + public ScheduledJobScope Scope { get; set; } = ScheduledJobScope.Global; + + /// Whether a new occurrence is skipped while a prior one is still running. Default SkipIfRunning. + public OverlapPolicy Overlap { get; set; } = OverlapPolicy.SkipIfRunning; + + /// How late a missed occurrence may still fire. Null uses the scheduler default. + public TimeSpan? MisfireWindow { get; set; } + + /// Maximum retry attempts for a failed occurrence. Default 3. + public int MaxRetries { get; set; } = 3; + + /// Whether the schedule is active. Default true. + public bool Enabled { get; set; } = true; + + /// Time zone the CRON expression is evaluated in. Null uses the scheduler default (UTC). + public TimeZoneInfo? TimeZone { get; set; } + + /// Typed arguments serialized into every occurrence's payload (see ). + public object? Arguments { get; set; } +} + +public interface IJobScheduler +{ + Task ScheduleAsync(ScheduledJobDefinition definition, CancellationToken cancellationToken = default); + Task UnscheduleAsync(string name, CancellationToken cancellationToken = default); + Task> GetSchedulesAsync(CancellationToken cancellationToken = default); +} + +public sealed class InMemoryJobScheduler : IJobScheduler +{ + private readonly ConcurrentDictionary _definitions = new(StringComparer.Ordinal); + + public Task ScheduleAsync(ScheduledJobDefinition definition, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(definition); + ArgumentException.ThrowIfNullOrEmpty(definition.Name); + ArgumentException.ThrowIfNullOrEmpty(definition.Cron); + cancellationToken.ThrowIfCancellationRequested(); + + if (definition.MaxRetries < 0) + throw new ArgumentOutOfRangeException(nameof(definition), definition.MaxRetries, "MaxRetries must be greater than or equal to zero."); + + if (definition.JobType is not null && !typeof(IJob).IsAssignableFrom(definition.JobType)) + throw new ArgumentException("JobType must implement IJob.", nameof(definition)); + + JobScheduleProcessor.ValidateCron(definition.Cron); + _definitions[definition.Name] = definition; + return Task.CompletedTask; + } + + public Task UnscheduleAsync(string name, CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrEmpty(name); + cancellationToken.ThrowIfCancellationRequested(); + _definitions.TryRemove(name, out _); + return Task.CompletedTask; + } + + public Task> GetSchedulesAsync(CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + return Task.FromResult>(_definitions.Values.OrderBy(d => d.Name, StringComparer.Ordinal).ToArray()); + } +} + +public sealed class JobScheduleProcessor +{ + private static readonly TimeSpan DefaultLease = TimeSpan.FromMinutes(5); + private static readonly TimeSpan DefaultMisfireWindow = TimeSpan.FromMinutes(1); + + private readonly IJobScheduler _scheduler; + private readonly IJobRuntimeStore _store; + private readonly IJobWorker _jobWorker; + private readonly TimeProvider _timeProvider; + private readonly IJobTypeRegistry _jobTypes; + private readonly ISerializer _serializer; + private readonly string _nodeId; + private readonly IMessageTransport? _transport; + + public JobScheduleProcessor(IJobScheduler scheduler, IJobRuntimeStore store, IJobWorker jobWorker, TimeProvider? timeProvider = null, string? nodeId = null, IMessageTransport? transport = null, IJobTypeRegistry? jobTypes = null, ISerializer? serializer = null) + { + _scheduler = scheduler ?? throw new ArgumentNullException(nameof(scheduler)); + _store = store ?? throw new ArgumentNullException(nameof(store)); + _jobWorker = jobWorker ?? throw new ArgumentNullException(nameof(jobWorker)); + _timeProvider = timeProvider ?? TimeProvider.System; + _jobTypes = jobTypes ?? new JobTypeRegistry(); + _serializer = serializer ?? DefaultSerializer.Instance; + _nodeId = !String.IsNullOrEmpty(nodeId) ? nodeId : NodeIdentity.Current; + _transport = transport; + } + + public Task> EnqueueDueOccurrencesAsync(CancellationToken cancellationToken = default) + { + return EnqueueDueOccurrencesAsync(_timeProvider.GetUtcNow(), cancellationToken); + } + + public async Task> EnqueueDueOccurrencesAsync(DateTimeOffset utcNow, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + var scheduled = new List(); + var definitions = await _scheduler.GetSchedulesAsync(cancellationToken).ConfigureAwait(false); + + foreach (var definition in definitions) + { + if (!definition.Enabled) + continue; + + var cron = ParseCron(definition.Cron); + var timeZone = definition.TimeZone ?? TimeZoneInfo.Utc; + var window = definition.MisfireWindow ?? DefaultMisfireWindow; + if (window < TimeSpan.Zero) + throw new ArgumentOutOfRangeException(nameof(definition), window, "MisfireWindow must be greater than or equal to zero."); + + string scopeKey = GetScopeKey(definition); + + // Materialize every occurrence that fell due within the misfire window, not just the most recent, so a + // scheduler that lagged behind the cadence does not silently drop intermediate ticks. Deterministic + // occurrence ids dedupe across overlapping windows and across nodes ticking simultaneously. + var occurrences = cron.GetOccurrences(utcNow - window, utcNow, timeZone, fromInclusive: true, toInclusive: true).ToList(); + if (occurrences.Count == 0) + continue; + + if (definition.Overlap == OverlapPolicy.SkipIfRunning) + { + // Don't stampede: if a prior occurrence is still pending or running, skip this tick entirely; + // otherwise collapse the window to a single (most recent) catch-up occurrence. + if (await HasActiveOccurrenceAsync(definition.Name, scopeKey, cancellationToken).ConfigureAwait(false)) + continue; + + occurrences = [occurrences[^1]]; + } + + foreach (var occurrence in occurrences) + { + string jobId = CreateOccurrenceId(definition.Name, occurrence, scopeKey); + + if (await _store.GetAsync(jobId, cancellationToken).ConfigureAwait(false) is not null) + continue; + + await _store.CreateIfAbsentAsync(new JobState + { + JobId = jobId, + Name = definition.Name, + JobType = GetJobTypeName(definition.JobType), + // Explicitly typed: the byte[] -> ReadOnlyMemory conversion maps a null array to an EMPTY memory, + // which would make an argless occurrence look like it carries a zero-byte payload. + Payload = definition.Arguments is null ? null : (ReadOnlyMemory?)_serializer.SerializeToBytes(definition.Arguments), + PayloadType = definition.Arguments?.GetType().FullName, + Status = JobStatus.Scheduled, + CreatedUtc = utcNow, + LastUpdatedUtc = utcNow, + ScheduledForUtc = occurrence + }, cancellationToken).ConfigureAwait(false); + + var dispatch = new ScheduledDispatchState + { + DispatchId = jobId, + Kind = ScheduledDispatchKind.JobOccurrence, + JobName = definition.Name, + Body = Array.Empty(), + Headers = CreateOccurrenceHeaders(definition, occurrence, scopeKey), + DueUtc = utcNow, + JobId = jobId + }; + + await _store.ScheduleDispatchAsync(dispatch, cancellationToken).ConfigureAwait(false); + scheduled.Add(dispatch); + } + } + + return scheduled; + } + + public Task RunDueOccurrencesAsync(CancellationToken cancellationToken = default) + { + return RunDueOccurrencesAsync(_timeProvider.GetUtcNow(), 100, null, cancellationToken); + } + + public async Task RunDueOccurrencesAsync(DateTimeOffset utcNow, int limit = 100, TimeSpan? lease = null, CancellationToken cancellationToken = default) + { + cancellationToken.ThrowIfCancellationRequested(); + + var definitions = (await _scheduler.GetSchedulesAsync(cancellationToken).ConfigureAwait(false)) + .ToDictionary(d => d.Name, StringComparer.Ordinal); + + var dispatches = await _store.ClaimDueDispatchesAsync(utcNow, limit, _nodeId, lease ?? DefaultLease, cancellationToken).ConfigureAwait(false); + int completed = 0; + + // Materialize delayed/scheduled MESSAGES before running any job occurrence: message dispatch is cheap and + // latency-sensitive (it is the messaging delayed-delivery fallback), so it must never wait behind a long job + // run that happened to be claimed earlier in the same batch. + foreach (var dispatch in dispatches) + { + if (dispatch.Kind is ScheduledDispatchKind.QueueMessage or ScheduledDispatchKind.PubSubMessage) + { + await MaterializeMessageDispatchAsync(dispatch, cancellationToken).ConfigureAwait(false); + completed++; + } + } + + foreach (var dispatch in dispatches) + { + if (dispatch.Kind is ScheduledDispatchKind.QueueMessage or ScheduledDispatchKind.PubSubMessage) + continue; + + if (dispatch.Kind != ScheduledDispatchKind.JobOccurrence) + { + await _store.ReleaseDispatchAsync(dispatch.DispatchId, _nodeId, utcNow.AddMinutes(1), cancellationToken).ConfigureAwait(false); + continue; + } + + if (dispatch.JobName is null || !definitions.TryGetValue(dispatch.JobName, out var definition) || !definition.Enabled || definition.JobType is null) + { + await _store.ReleaseDispatchAsync(dispatch.DispatchId, _nodeId, utcNow.AddMinutes(1), cancellationToken).ConfigureAwait(false); + continue; + } + + string jobId = dispatch.JobId ?? dispatch.DispatchId; + + try + { + if (!await TryPrepareOccurrenceForRunAsync(jobId, definition, utcNow, cancellationToken).ConfigureAwait(false)) + { + // Retire (don't reschedule) the dispatch when the occurrence has reached a terminal state — e.g. it + // was dead-lettered in TryPrepareOccurrenceForRunAsync, or a worker completed it but crashed before + // CompleteDispatchAsync. Otherwise a terminal occurrence's dispatch would be re-claimed forever. + var pending = await _store.GetAsync(jobId, cancellationToken).ConfigureAwait(false); + if (pending is { Status: JobStatus.Completed or JobStatus.Cancelled or JobStatus.DeadLettered }) + await _store.CompleteDispatchAsync(dispatch.DispatchId, _nodeId, cancellationToken).ConfigureAwait(false); + else + await _store.ReleaseDispatchAsync(dispatch.DispatchId, _nodeId, utcNow.AddMinutes(1), cancellationToken).ConfigureAwait(false); + continue; + } + + await _jobWorker.RunAsync(jobId, cancellationToken).ConfigureAwait(false); + + var state = await _store.GetAsync(jobId, cancellationToken).ConfigureAwait(false); + if (state?.Status == JobStatus.Failed) + { + if (state.Attempt <= definition.MaxRetries) + { + await _store.TryTransitionAsync(jobId, JobStatus.Failed, JobStatus.Scheduled, new JobStatePatch + { + ClearNodeId = true, + ClearLeaseExpiresUtc = true, + LastUpdatedUtc = utcNow + }, cancellationToken: cancellationToken).ConfigureAwait(false); + await _store.ReleaseDispatchAsync(dispatch.DispatchId, _nodeId, utcNow.Add(GetRetryBackoff(definition, state.Attempt)), cancellationToken).ConfigureAwait(false); + continue; + } + + await _store.TryTransitionAsync(jobId, JobStatus.Failed, JobStatus.DeadLettered, new JobStatePatch + { + ClearNodeId = true, + ClearLeaseExpiresUtc = true, + LastUpdatedUtc = utcNow + }, cancellationToken: cancellationToken).ConfigureAwait(false); + } + + await _store.CompleteDispatchAsync(dispatch.DispatchId, _nodeId, cancellationToken).ConfigureAwait(false); + completed++; + } + catch + { + await _store.ReleaseDispatchAsync(dispatch.DispatchId, _nodeId, utcNow.AddMinutes(1), CancellationToken.None).ConfigureAwait(false); + throw; + } + } + + return completed; + } + + private async Task MaterializeMessageDispatchAsync(ScheduledDispatchState dispatch, CancellationToken cancellationToken) + { + if (_transport is null) + throw new InvalidOperationException("A message transport is required to materialize scheduled queue and pub/sub dispatches."); + + if (dispatch.Destination is null) + throw new InvalidOperationException($"Scheduled {dispatch.Kind} dispatch \"{dispatch.DispatchId}\" has no destination address."); + + await _transport.SendAsync(dispatch.Destination, [ + new TransportMessage + { + MessageId = dispatch.DispatchId, + Body = dispatch.Body, + Headers = dispatch.Headers + } + ], dispatch.Options with { DeliverAt = null }, cancellationToken).ConfigureAwait(false); + + // SendAsync is throw-on-failure; reaching here means the dispatch was materialized, so retire it. + await _store.CompleteDispatchAsync(dispatch.DispatchId, _nodeId, cancellationToken).ConfigureAwait(false); + } + + private async Task TryPrepareOccurrenceForRunAsync(string jobId, ScheduledJobDefinition definition, DateTimeOffset utcNow, CancellationToken cancellationToken) + { + if (await _store.TryTransitionAsync(jobId, JobStatus.Scheduled, JobStatus.Queued, new JobStatePatch { JobType = GetJobTypeName(definition.JobType), LastUpdatedUtc = utcNow }, cancellationToken: cancellationToken).ConfigureAwait(false)) + return true; + + var state = await _store.GetAsync(jobId, cancellationToken).ConfigureAwait(false); + if (state?.Status != JobStatus.Processing || state.LeaseExpiresUtc is null || state.LeaseExpiresUtc > utcNow) + return false; + + if (state.Attempt > definition.MaxRetries) + { + await _store.TryTransitionAsync(jobId, JobStatus.Processing, JobStatus.DeadLettered, new JobStatePatch + { + ClearNodeId = true, + ClearLeaseExpiresUtc = true, + LastUpdatedUtc = utcNow + }, cancellationToken: cancellationToken).ConfigureAwait(false); + return false; + } + + return await _store.TryTransitionAsync(jobId, JobStatus.Processing, JobStatus.Queued, new JobStatePatch + { + JobType = GetJobTypeName(definition.JobType), + ClearNodeId = true, + ClearLeaseExpiresUtc = true, + LastUpdatedUtc = utcNow + }, cancellationToken: cancellationToken).ConfigureAwait(false); + } + + private string? GetJobTypeName(Type? jobType) + { + return jobType is null ? null : _jobTypes.GetName(jobType); + } + + private static TimeSpan GetRetryBackoff(ScheduledJobDefinition definition, int attempt) + { + if (definition.RetryBackoff is { } custom) + return custom(attempt); + + // Capped exponential backoff: 1s, 2s, 4s, ... up to 5 minutes. + double seconds = Math.Min(300, Math.Pow(2, Math.Max(0, attempt - 1))); + return TimeSpan.FromSeconds(seconds); + } + + private async Task HasActiveOccurrenceAsync(string name, string scopeKey, CancellationToken cancellationToken) + { + var states = await _store.QueryAsync(new JobQuery { Name = name, Limit = 1000 }, cancellationToken).ConfigureAwait(false); + return states.Any(s => OccurrenceMatchesScope(s.JobId, name, scopeKey) && s.Status is JobStatus.Queued or JobStatus.Scheduled or JobStatus.Processing); + } + + // Exact scope match, not a JobId suffix test: an occurrence id is "{name}:{14-digit-timestamp}:{scopeKey}", and a + // scope key (a node id) can itself contain ':' (NodeIdentity.Current is "{machine}:{pid}:{token}"), so a naive + // EndsWith(":{scopeKey}") would let one node's occurrence count as another's. The query is already filtered to this + // name, so strip the literal "{name}:" prefix and the fixed-width timestamp, then compare the remainder exactly. + private static bool OccurrenceMatchesScope(string jobId, string name, string scopeKey) + { + string prefix = $"{name}:"; + if (!jobId.StartsWith(prefix, StringComparison.Ordinal)) + return false; + + var rest = jobId.AsSpan(prefix.Length); + return rest.Length >= 15 && rest[14] == ':' && rest[15..].SequenceEqual(scopeKey); + } + + private string GetScopeKey(ScheduledJobDefinition definition) + { + return definition.Scope == ScheduledJobScope.PerNode ? _nodeId : "global"; + } + + private static string CreateOccurrenceId(string name, DateTimeOffset scheduledForUtc, string scopeKey) + { + return $"{name}:{scheduledForUtc.UtcDateTime:yyyyMMddHHmmss}:{scopeKey}"; + } + + private static MessageHeaders CreateOccurrenceHeaders(ScheduledJobDefinition definition, DateTimeOffset scheduledForUtc, string scopeKey) + { + return MessageHeaders.Create([ + new KeyValuePair("job.name", definition.Name), + new KeyValuePair("job.scheduled_for", scheduledForUtc.UtcDateTime.ToString("O")), + new KeyValuePair("job.scope", scopeKey) + ]); + } + + internal static void ValidateCron(string expression) + { + ParseCron(expression); + } + + /// + /// Parses a 5- or 6-field cron expression using the vendored Cronos parser. Six fields are interpreted as + /// seconds-first (); five fields use the standard format. Cronos + /// supports the full grammar (ranges, steps, lists, L/W/#, named months/days, and macros + /// such as @daily). + /// + private static CronExpression ParseCron(string expression) + { + ArgumentException.ThrowIfNullOrWhiteSpace(expression); + + if (expression.StartsWith('@')) + return CronExpression.Parse(expression, CronFormat.IncludeSeconds); + + int fieldCount = expression.Split(' ', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries).Length; + var format = fieldCount == 6 ? CronFormat.IncludeSeconds : CronFormat.Standard; + return CronExpression.Parse(expression, format); + } +} diff --git a/src/Foundatio/Jobs/JobWithLockBase.cs b/src/Foundatio/Jobs/JobWithLockBase.cs index 8af43866d..44d9af51a 100644 --- a/src/Foundatio/Jobs/JobWithLockBase.cs +++ b/src/Foundatio/Jobs/JobWithLockBase.cs @@ -7,7 +7,7 @@ using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; -namespace Foundatio.Jobs; +namespace Foundatio.Jobs.Legacy; public abstract class JobWithLockBase : IJobWithOptions, IHaveLogger, IHaveLoggerFactory, IHaveTimeProvider, IHaveResiliencePolicyProvider { diff --git a/src/Foundatio/Jobs/LegacyJob.cs b/src/Foundatio/Jobs/LegacyJob.cs new file mode 100644 index 000000000..0d41e3b40 --- /dev/null +++ b/src/Foundatio/Jobs/LegacyJob.cs @@ -0,0 +1,14 @@ +using System.Threading; +using System.Threading.Tasks; + +namespace Foundatio.Jobs.Legacy; + +/// +/// The legacy job contract, run once or continuously by the legacy and hosted runners. +/// Superseded by the durable-runtime (which is handed a +/// per run); kept for compatibility. +/// +public interface IJob +{ + Task RunAsync(CancellationToken cancellationToken = default); +} diff --git a/src/Foundatio/Jobs/LegacyJobResult.cs b/src/Foundatio/Jobs/LegacyJobResult.cs new file mode 100644 index 000000000..850e7b5ca --- /dev/null +++ b/src/Foundatio/Jobs/LegacyJobResult.cs @@ -0,0 +1,85 @@ +using System; +using Microsoft.Extensions.Logging; + +namespace Foundatio.Jobs.Legacy; + +public class JobResult +{ + public bool IsCancelled { get; set; } + public Exception? Error { get; set; } + public string Message { get; set; } = String.Empty; + public bool IsSuccess { get; set; } + + public static readonly JobResult None = new() + { + IsSuccess = true + }; + + public static readonly JobResult Cancelled = new() + { + IsCancelled = true + }; + + public static readonly JobResult Success = new() + { + IsSuccess = true + }; + + public static JobResult FromException(Exception exception, string? message = null) + { + return new JobResult + { + Error = exception, + IsSuccess = false, + Message = message ?? exception.Message + }; + } + + public static JobResult CancelledWithMessage(string message) + { + return new JobResult + { + IsCancelled = true, + Message = message + }; + } + + public static JobResult SuccessWithMessage(string message) + { + return new JobResult + { + IsSuccess = true, + Message = message + }; + } + + public static JobResult FailedWithMessage(string message) + { + return new JobResult + { + IsSuccess = false, + Message = message + }; + } +} + +public static class JobResultExtensions +{ + public static void LogJobResult(this ILogger logger, JobResult result, string? jobName) + { + if (result is null) + { + logger.LogError("Null job run result for {JobName}", jobName); + return; + } + + if (result.IsCancelled) + logger.LogWarning(result.Error, "Job run {JobName} cancelled: {Message}", jobName, result.Message); + else if (!result.IsSuccess) + logger.LogError(result.Error, "Job run {JobName} failed: {Message}", jobName, result.Message); + else if (!String.IsNullOrEmpty(result.Message)) + logger.LogInformation("Job run {JobName} succeeded: {Message}", jobName, result.Message); + else + logger.LogDebug("Job run {JobName} succeeded", jobName); + } +} diff --git a/src/Foundatio/Jobs/LegacyJobRunExtensions.cs b/src/Foundatio/Jobs/LegacyJobRunExtensions.cs new file mode 100644 index 000000000..d42564a3e --- /dev/null +++ b/src/Foundatio/Jobs/LegacyJobRunExtensions.cs @@ -0,0 +1,134 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Utility; +using Microsoft.Extensions.Logging; + +namespace Foundatio.Jobs.Legacy; + +/// +/// A job that exposes configurable options for execution behavior. +/// +public interface IJobWithOptions : IJob +{ + /// + /// Gets or sets the options controlling job execution (name, interval, iteration limit). + /// + JobOptions? Options { get; set; } +} + +public static class LegacyJobExtensions +{ + /// + /// Runs the job, converting cancellation and unhandled exceptions into a instead of throwing. + /// + public static async Task TryRunAsync(this IJob job, CancellationToken cancellationToken = default) + { + try + { + return await job.RunAsync(cancellationToken).AnyContext(); + } + catch (OperationCanceledException) + { + return JobResult.Cancelled; + } + catch (Exception ex) + { + return JobResult.FromException(ex); + } + } + + /// + /// Runs the job continuously until the cancellation token is set or the iteration limit is reached. + /// + /// Returns the iteration count for normal jobs. For queue-based jobs this will be the number of items processed successfully. + public static Task RunContinuousAsync(this IJob job, TimeSpan? interval = null, int iterationLimit = -1, + CancellationToken cancellationToken = default, Func>? continuationCallback = null) + { + var options = JobOptions.GetDefaults(job); + options.Interval = interval; + options.IterationLimit = iterationLimit; + return RunContinuousAsync(job, options, cancellationToken, continuationCallback); + } + + /// + /// Runs the job continuously until the cancellation token is set or the iteration limit is reached. + /// + /// Returns the iteration count for normal jobs. For queue based jobs this will be the amount of items processed successfully. + public static async Task RunContinuousAsync(this IJob job, JobOptions options, CancellationToken cancellationToken = default, Func>? continuationCallback = null) + { + int iterations = 0; + var logger = job.GetLogger(); + + int queueItemsProcessed = 0; + bool isQueueJob = job.GetType().GetInterfaces().Any(x => x.IsGenericType && x.GetGenericTypeDefinition() == typeof(IQueueJob<>)); + + string jobId = Guid.NewGuid().ToString("N").Substring(0, 10); + using var jobScope = logger.BeginScope(s => s.Property("job.name", options.Name ?? String.Empty).Property("job.id", jobId)); + logger.LogInformation("Starting continuous job type {JobName} on machine {MachineName}...", options.Name, Environment.MachineName); + + while (!cancellationToken.IsCancellationRequested) + { + using var activity = FoundatioDiagnostics.ActivitySource.StartActivity($"Job: {options.Name}"); + + string jobRunId = Guid.NewGuid().ToString("N").Substring(0, 10); + using var _ = logger.BeginScope(s => s.Property("job.run_id", jobRunId)); + var result = await job.TryRunAsync(cancellationToken).AnyContext(); + logger.LogJobResult(result, options.Name); + + iterations++; + if (isQueueJob && result.IsSuccess) + queueItemsProcessed++; + + if (cancellationToken.IsCancellationRequested || (options.IterationLimit > -1 && options.IterationLimit <= iterations)) + break; + + if (result.Error != null) + { + await job.GetTimeProvider().SafeDelay(TimeSpan.FromMilliseconds(Math.Max((int)(options.Interval?.TotalMilliseconds ?? 0), 100)), cancellationToken).AnyContext(); + } + else if (options.Interval.HasValue && options.Interval.Value > TimeSpan.Zero) + { + await job.GetTimeProvider().SafeDelay(options.Interval.Value, cancellationToken).AnyContext(); + } + + // needed to yield back a task for jobs that aren't async + await Task.Yield(); + + if (cancellationToken.IsCancellationRequested) + break; + + if (continuationCallback is null) + continue; + + try + { + if (!await continuationCallback().AnyContext()) + break; + } + catch (Exception ex) + { + logger.LogError(ex, "Error in continuation callback: {Message}", ex.Message); + } + } + + if (cancellationToken.IsCancellationRequested) + logger.LogTrace("Job cancellation requested"); + + if (options.IterationLimit > 0) + { + logger.LogInformation( + "Stopping continuous job type {JobName} on machine {MachineName}: Job ran {Iterations} times (Limit={IterationLimit})", + options.Name, Environment.MachineName, iterations, options.IterationLimit); + } + else + { + logger.LogInformation( + "Stopping continuous job type {JobName} on machine {MachineName}: Job ran {Iterations} times", + options.Name, Environment.MachineName, iterations); + } + + return isQueueJob ? queueItemsProcessed : iterations; + } +} diff --git a/src/Foundatio/Jobs/QueueEntryContext.cs b/src/Foundatio/Jobs/QueueEntryContext.cs index 04ba268a8..dce43f5ef 100644 --- a/src/Foundatio/Jobs/QueueEntryContext.cs +++ b/src/Foundatio/Jobs/QueueEntryContext.cs @@ -4,7 +4,7 @@ using Foundatio.Queues; using Foundatio.Utility; -namespace Foundatio.Jobs; +namespace Foundatio.Jobs.Legacy; public class QueueEntryContext : JobContext where T : class { diff --git a/src/Foundatio/Jobs/QueueJobBase.cs b/src/Foundatio/Jobs/QueueJobBase.cs index fb1f520d4..e0f451eaa 100644 --- a/src/Foundatio/Jobs/QueueJobBase.cs +++ b/src/Foundatio/Jobs/QueueJobBase.cs @@ -9,7 +9,7 @@ using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; -namespace Foundatio.Jobs; +namespace Foundatio.Jobs.Legacy; public abstract class QueueJobBase : IQueueJob, IHaveLogger, IHaveLoggerFactory, IHaveTimeProvider, IHaveResiliencePolicyProvider where T : class { diff --git a/src/Foundatio/Jobs/WorkItemJob/WorkItemContext.cs b/src/Foundatio/Jobs/WorkItemJob/WorkItemContext.cs index 355d779ed..7d043ef8a 100644 --- a/src/Foundatio/Jobs/WorkItemJob/WorkItemContext.cs +++ b/src/Foundatio/Jobs/WorkItemJob/WorkItemContext.cs @@ -3,7 +3,7 @@ using System.Threading.Tasks; using Foundatio.Lock; -namespace Foundatio.Jobs; +namespace Foundatio.Jobs.Legacy; public class WorkItemContext { diff --git a/src/Foundatio/Jobs/WorkItemJob/WorkItemData.cs b/src/Foundatio/Jobs/WorkItemJob/WorkItemData.cs index 987ee5d32..d0004930e 100644 --- a/src/Foundatio/Jobs/WorkItemJob/WorkItemData.cs +++ b/src/Foundatio/Jobs/WorkItemJob/WorkItemData.cs @@ -1,7 +1,7 @@ using Foundatio.Metrics; using Foundatio.Queues; -namespace Foundatio.Jobs; +namespace Foundatio.Jobs.Legacy; public class WorkItemData : IHaveSubMetricName, IHaveUniqueIdentifier { diff --git a/src/Foundatio/Jobs/WorkItemJob/WorkItemHandlers.cs b/src/Foundatio/Jobs/WorkItemJob/WorkItemHandlers.cs index 0b9082673..48158268c 100644 --- a/src/Foundatio/Jobs/WorkItemJob/WorkItemHandlers.cs +++ b/src/Foundatio/Jobs/WorkItemJob/WorkItemHandlers.cs @@ -8,7 +8,7 @@ using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; -namespace Foundatio.Jobs; +namespace Foundatio.Jobs.Legacy; public class WorkItemHandlers { diff --git a/src/Foundatio/Jobs/WorkItemJob/WorkItemJob.cs b/src/Foundatio/Jobs/WorkItemJob/WorkItemJob.cs index 5d37547e6..12632fdc3 100644 --- a/src/Foundatio/Jobs/WorkItemJob/WorkItemJob.cs +++ b/src/Foundatio/Jobs/WorkItemJob/WorkItemJob.cs @@ -3,14 +3,14 @@ using System.Diagnostics; using System.Threading; using System.Threading.Tasks; -using Foundatio.Messaging; +using Foundatio.Messaging.Legacy; using Foundatio.Queues; using Foundatio.Serializer; using Foundatio.Utility; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; -namespace Foundatio.Jobs; +namespace Foundatio.Jobs.Legacy; [Job(Description = "Processes adhoc work item queues entries")] public class WorkItemJob : IQueueJob, IHaveLogger, IHaveLoggerFactory diff --git a/src/Foundatio/Jobs/WorkItemJob/WorkItemQueueExtensions.cs b/src/Foundatio/Jobs/WorkItemJob/WorkItemQueueExtensions.cs index 72f3704b8..c89dbcf91 100644 --- a/src/Foundatio/Jobs/WorkItemJob/WorkItemQueueExtensions.cs +++ b/src/Foundatio/Jobs/WorkItemJob/WorkItemQueueExtensions.cs @@ -5,7 +5,7 @@ using Foundatio.Serializer; using Foundatio.Utility; -namespace Foundatio.Jobs; +namespace Foundatio.Jobs.Legacy; public static class WorkItemQueueExtensions { diff --git a/src/Foundatio/Jobs/WorkItemJob/WorkItemStatus.cs b/src/Foundatio/Jobs/WorkItemJob/WorkItemStatus.cs index 1ddad0248..433858b09 100644 --- a/src/Foundatio/Jobs/WorkItemJob/WorkItemStatus.cs +++ b/src/Foundatio/Jobs/WorkItemJob/WorkItemStatus.cs @@ -1,4 +1,4 @@ -namespace Foundatio.Jobs; +namespace Foundatio.Jobs.Legacy; public class WorkItemStatus { diff --git a/src/Foundatio/Lock/CacheLockProvider.cs b/src/Foundatio/Lock/CacheLockProvider.cs index aae3d4d14..d48b7bb14 100644 --- a/src/Foundatio/Lock/CacheLockProvider.cs +++ b/src/Foundatio/Lock/CacheLockProvider.cs @@ -6,7 +6,7 @@ using System.Threading.Tasks; using Foundatio.AsyncEx; using Foundatio.Caching; -using Foundatio.Messaging; +using Foundatio.Messaging.Legacy; using Foundatio.Resilience; using Foundatio.Utility; using Microsoft.Extensions.Logging; diff --git a/src/Foundatio/Messaging/IMessageBus.cs b/src/Foundatio/Messaging/IMessageBus.cs index 1567da0d5..90cc8e056 100644 --- a/src/Foundatio/Messaging/IMessageBus.cs +++ b/src/Foundatio/Messaging/IMessageBus.cs @@ -2,7 +2,7 @@ using System.Collections.Generic; using System.Diagnostics.CodeAnalysis; -namespace Foundatio.Messaging; +namespace Foundatio.Messaging.Legacy; /// /// Represents a message bus that supports both publishing and subscribing to messages. diff --git a/src/Foundatio/Messaging/IMessageContext.cs b/src/Foundatio/Messaging/IMessageContext.cs new file mode 100644 index 000000000..4f3fee070 --- /dev/null +++ b/src/Foundatio/Messaging/IMessageContext.cs @@ -0,0 +1,136 @@ +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace Foundatio.Messaging; + +public enum AckMode +{ + Auto, + Manual +} + +/// +/// Core-owned retry and dead-letter policy. Foundatio always owns redelivery and dead-lettering so the behavior is +/// identical across transports; transports stay simple and only provide the underlying primitives (redelivery and an +/// optional dead-letter sink). Configure a default on ; a subscription can override +/// /backoff per subscription. +/// +public sealed record RetryPolicy +{ + /// Maximum delivery attempts for a failing handler before the message is dead-lettered. Default 5. + public int MaxAttempts { get; init; } = 5; + + /// + /// Delay before each redelivery given the 1-based attempt number. Defaults to + /// (immediate first retry, then 10s/20s/30s with jitter). Set null to defer to the transport's own redelivery timing. + /// + public Func? Backoff { get; init; } = DefaultBackoff; + + /// + /// Marks a handler failure as unrecoverable: when the predicate returns true the message is dead-lettered + /// immediately instead of retried (a poison message should not burn its attempt budget). Deserialization failures + /// are always unrecoverable regardless of this predicate. A subscription's + /// overrides this default. + /// + public Func? DeadLetterWhen { get; init; } + + /// + /// Destination terminal (dead-lettered) messages are sent to when the transport has no native dead-letter sink. + /// Null (default) derives "{source}.deadletter" per source. Ignored when the transport supports native dead-lettering. + /// + public string? DeadLetterDestination { get; init; } + + /// + /// The default redelivery curve: an immediate first retry, then 10s/20s/30s (capped) with ±20% jitter — the delay + /// shape mature messaging stacks converged on. The attempt number is 1-based: the value is the delay applied after + /// that attempt failed. + /// + public static readonly Func DefaultBackoff = attempt => + { + if (attempt <= 1) + return TimeSpan.Zero; + + double seconds = Math.Min((attempt - 1) * 10, 30); + double jitter = 1 + (Random.Shared.NextDouble() * 0.4 - 0.2); + return TimeSpan.FromSeconds(seconds * jitter); + }; + + /// Maximum attempts for a message whose type has no registered consumer before it is dead-lettered as "no-handler". Default 50. + public int UnmatchedMaxAttempts { get; init; } = 50; + + /// Delay before redelivering an unmatched-type message. Null defers to the transport's own redelivery timing. + public Func? UnmatchedBackoff { get; init; } +} + +/// +/// Thrown by the consumer loop when a message arrives on a shared destination whose type has no registered consumer +/// on this node (for example a newer message type mid rolling-deploy, or a misconfiguration). It is surfaced loudly +/// per message and isolated to that message — the receive loop and the other type handlers keep running. +/// +public sealed class UnhandledMessageTypeException : Exception +{ + public UnhandledMessageTypeException(string? messageType, string source) + : base($"No consumer is registered for message type \"{messageType ?? "(unknown)"}\" received on source \"{source}\".") + { + MessageType = messageType; + SourceName = source; + } + + public string? MessageType { get; } + public string SourceName { get; } +} + +public sealed record RejectOptions +{ + /// + /// When false (default) the message is returned for redelivery (a retry). When true the message is terminal: it + /// is moved to the transport's native dead-letter sink where one exists, otherwise sent to the configured or + /// derived ("{source}.deadletter") dead-letter destination. Terminal messages are never redelivered. + /// + public bool Terminal { get; init; } + + /// Reason carried to the dead-letter sink for a terminal reject. + public string? Reason { get; init; } + + /// + /// The failure behind a terminal reject; its type/message/stack are stamped as forensics headers on the + /// dead-lettered message so a dead message is triageable with plain transport tooling. + /// + public Exception? Exception { get; init; } + + /// + /// An explicit delay before the message is redelivered. Honored only for a non-terminal reject, served natively + /// when the transport supports redelivery delay within its advertised maximum, otherwise through the runtime store. + /// When null the transport's own redelivery timing applies. + /// + public TimeSpan? RedeliveryDelay { get; init; } + + /// + /// When true, a the transport cannot honor (no native support and no runtime store) + /// degrades to immediate redelivery instead of failing. The core's retry policy rejects with best-effort delays; + /// an explicit caller-specified delay defaults to strict. + /// + public bool BestEffortDelay { get; init; } +} + +public interface IMessageContext +{ + string Id { get; } + ReadOnlyMemory Body { get; } + MessageHeaders Headers { get; } + string? CorrelationId { get; } + string? MessageType { get; } + MessagePriority Priority { get; } + int Attempts { get; } + bool IsHandled { get; } + CancellationToken CancellationToken { get; } + Task CompleteAsync(CancellationToken cancellationToken = default); + Task RejectAsync(RejectOptions? options = null, CancellationToken cancellationToken = default); + Task RenewLockAsync(TimeSpan? duration = null, CancellationToken cancellationToken = default); +} + +public interface IMessageContext : IMessageContext where T : class +{ + T Message { get; } +} diff --git a/src/Foundatio/Messaging/IMessageHandler.cs b/src/Foundatio/Messaging/IMessageHandler.cs new file mode 100644 index 000000000..196095076 --- /dev/null +++ b/src/Foundatio/Messaging/IMessageHandler.cs @@ -0,0 +1,18 @@ +using System.Threading; +using System.Threading.Tasks; + +namespace Foundatio.Messaging; + +/// +/// Handles messages of type . Register with +/// AddFoundatio().Messaging.AddHandler<T, THandler>() — registration carries no topology decision; the +/// caller's verb on decides delivery (SendAsync = one handler instance across the +/// fleet, PublishAsync = once per subscribing service, or every instance with +/// ). A hosted service starts and dispatches to it. Handlers are +/// resolved from DI in their own scope per message, so they can inject scoped dependencies. Throwing from +/// triggers the core's retry/dead-letter policy. +/// +public interface IMessageHandler where T : class +{ + Task HandleAsync(IMessageContext context, CancellationToken cancellationToken); +} diff --git a/src/Foundatio/Messaging/IMessagePublisher.cs b/src/Foundatio/Messaging/IMessagePublisher.cs index 1961d85f0..531a4efcd 100644 --- a/src/Foundatio/Messaging/IMessagePublisher.cs +++ b/src/Foundatio/Messaging/IMessagePublisher.cs @@ -2,7 +2,7 @@ using System.Threading; using System.Threading.Tasks; -namespace Foundatio.Messaging; +namespace Foundatio.Messaging.Legacy; /// /// Publishes messages to all subscribers listening for the message type. diff --git a/src/Foundatio/Messaging/IMessageSubscriber.cs b/src/Foundatio/Messaging/IMessageSubscriber.cs index a5c107d01..0d2ec7279 100644 --- a/src/Foundatio/Messaging/IMessageSubscriber.cs +++ b/src/Foundatio/Messaging/IMessageSubscriber.cs @@ -2,7 +2,7 @@ using System.Threading; using System.Threading.Tasks; -namespace Foundatio.Messaging; +namespace Foundatio.Messaging.Legacy; /// /// Subscribes to messages published on the message bus. diff --git a/src/Foundatio/Messaging/InMemoryMessageBus.cs b/src/Foundatio/Messaging/InMemoryMessageBus.cs index 7cd2132e4..a7a511f1c 100644 --- a/src/Foundatio/Messaging/InMemoryMessageBus.cs +++ b/src/Foundatio/Messaging/InMemoryMessageBus.cs @@ -6,7 +6,7 @@ using Foundatio.Utility; using Microsoft.Extensions.Logging; -namespace Foundatio.Messaging; +namespace Foundatio.Messaging.Legacy; public class InMemoryMessageBus : MessageBusBase { diff --git a/src/Foundatio/Messaging/InMemoryMessageBusOptions.cs b/src/Foundatio/Messaging/InMemoryMessageBusOptions.cs index 1387eee31..7a7433614 100644 --- a/src/Foundatio/Messaging/InMemoryMessageBusOptions.cs +++ b/src/Foundatio/Messaging/InMemoryMessageBusOptions.cs @@ -1,4 +1,4 @@ -namespace Foundatio.Messaging; +namespace Foundatio.Messaging.Legacy; public class InMemoryMessageBusOptions : SharedMessageBusOptions { } diff --git a/src/Foundatio/Messaging/InMemoryMessageTransport.cs b/src/Foundatio/Messaging/InMemoryMessageTransport.cs new file mode 100644 index 000000000..16d2b28cd --- /dev/null +++ b/src/Foundatio/Messaging/InMemoryMessageTransport.cs @@ -0,0 +1,829 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; +using System.Threading.Channels; +using Foundatio.AsyncEx; +using Foundatio.Utility; + +namespace Foundatio.Messaging; + +public sealed class InMemoryMessageTransport : IMessageTransport, ISupportsPull, ISupportsPush, ISupportsVisibilityTimeout, ISupportsDeadLetter, ISupportsRedeliveryDelay, ISupportsLockRenewal, ISupportsStats, ISupportsProvisioning, ITransportInfo +{ + private static readonly TimeSpan _defaultLockRenewal = TimeSpan.FromMinutes(1); + + // Priority and expiration are honored on every role; there is no native delayed delivery (delays route through + // the runtime-store fallback) and no broker-imposed size or batch limits. + private static readonly TransportCapabilities _capabilities = new() + { + Priority = true, + Expiration = true, + Ordering = OrderingGuarantee.Fifo + }; + + private static readonly IReadOnlySet _supportedRoles = new HashSet + { + DestinationRole.Queue, + DestinationRole.Topic, + DestinationRole.Subscription, + DestinationRole.Binding + }; + + private readonly ConcurrentDictionary _destinations = new(StringComparer.OrdinalIgnoreCase); + private readonly ConcurrentDictionary _roles = new(StringComparer.OrdinalIgnoreCase); + private readonly ConcurrentDictionary> _topicSubscriptions = new(StringComparer.OrdinalIgnoreCase); + private readonly ConcurrentDictionary _redeliveryTimers = new(); + private readonly TimeProvider _timeProvider; + private readonly CancellationTokenSource _disposeCancellationTokenSource = new(); + private int _isDisposed; + + public InMemoryMessageTransport(TimeProvider? timeProvider = null) + { + _timeProvider = timeProvider ?? TimeProvider.System; + } + + public DeliveryGuarantee DeliveryGuarantee => DeliveryGuarantee.AtLeastOnce; + public IReadOnlySet SupportedRoles => _supportedRoles; + + public TransportCapabilities GetCapabilities(DestinationRole role) => _capabilities; + + // The in-memory transport has no broker-imposed ceiling on visibility or redelivery delay. + public TimeSpan? MaxVisibilityTimeout => null; + public TimeSpan? MaxRedeliveryDelay => null; + + public Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(destination); + ArgumentNullException.ThrowIfNull(messages); + + if (options.DeliverAt is { } deliverAt && deliverAt > _timeProvider.GetUtcNow()) + throw new NotSupportedException($"Transport \"{GetType().Name}\" does not support native delayed delivery. Use the runtime-store scheduled dispatch fallback."); + + // The address role picks the physical namespace, so a queue and a topic can share a route name (a message + // type that is both sent and published) without colliding or cross-delivering. + string key = StorageKey(destination); + + var results = new SendItemResult[messages.Count]; + for (int index = 0; index < messages.Count; index++) + { + var message = messages[index]; + // Each message gets a unique id so per-message settlement never aliases across distinct messages. + string messageId = message.MessageId ?? Guid.NewGuid().ToString("N"); + var stored = CreateStoredMessage(key, messageId, message, options); + EnqueueForDestination(key, stored); + + results[index] = new SendItemResult { MessageId = messageId }; + } + + return Task.FromResult(new SendResult { Items = results }); + } + + public Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct) + { + return ReceiveAsync(source, request, visibility: null, ct); + } + + public async Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, TimeSpan visibility, CancellationToken ct) + { + return await ReceiveAsync(source, request, (TimeSpan?)visibility, ct).AnyContext(); + } + + private async Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, TimeSpan? visibility, CancellationToken ct) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(source); + + int maxMessages = request.MaxMessages <= 0 ? 1 : request.MaxMessages; + var state = GetOrAddDestination(ReceivableKey(source)); + var entries = new List(maxMessages); + DateTimeOffset? waitUntil = request.MaxWaitTime is { } waitTime && waitTime > TimeSpan.Zero + ? _timeProvider.GetUtcNow().Add(waitTime) + : null; + + // Return any messages whose visibility window lapsed (consumer crashed without settling) to the queue so + // they are redelivered with an incremented delivery count — honoring the advertised at-least-once contract. + state.ReclaimExpired(_timeProvider.GetUtcNow()); + + while (entries.Count < maxMessages) + { + if (TryReceive(source, state, visibility, out var entry)) + { + entries.Add(entry); + continue; + } + + if (entries.Count > 0 || waitUntil is null) + break; + + TimeSpan remaining = waitUntil.Value - _timeProvider.GetUtcNow(); + if (remaining <= TimeSpan.Zero) + break; + + using var waitCancellationTokenSource = CancellationTokenSource.CreateLinkedTokenSource(ct, _disposeCancellationTokenSource.Token); + waitCancellationTokenSource.CancelAfter(remaining); + + try + { + if (!await state.WaitToReadAsync(waitCancellationTokenSource.Token).ConfigureAwait(false)) + break; + } + catch (OperationCanceledException) when (!ct.IsCancellationRequested && !_disposeCancellationTokenSource.IsCancellationRequested) + { + break; + } + } + + return entries; + } + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(entry); + + var receipt = GetReceipt(entry); + var state = GetExistingDestination(receipt.Destination); + + if (!state.InFlight.TryRemove(receipt.LockToken, out var inFlight) || !String.Equals(inFlight.Message.Id, entry.Id, StringComparison.Ordinal)) + throw new ReceiptExpiredException(); + + Interlocked.Increment(ref state.Completed); + return Task.CompletedTask; + } + + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(entry); + + var receipt = GetReceipt(entry); + var state = GetExistingDestination(receipt.Destination); + + if (!state.InFlight.TryRemove(receipt.LockToken, out var inFlight) || !String.Equals(inFlight.Message.Id, entry.Id, StringComparison.Ordinal)) + throw new ReceiptExpiredException(); + + Interlocked.Increment(ref state.Abandoned); + var redelivered = inFlight.Message with { DeliveryCount = entry.DeliveryCount + 1 }; + EnqueueStoredMessage(receipt.Destination, redelivered); + + return Task.CompletedTask; + } + + public Task AbandonAsync(TransportEntry entry, TimeSpan redeliveryDelay, CancellationToken ct) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(entry); + + if (redeliveryDelay <= TimeSpan.Zero) + return AbandonAsync(entry, ct); + + var receipt = GetReceipt(entry); + var state = GetExistingDestination(receipt.Destination); + + if (!state.InFlight.TryRemove(receipt.LockToken, out var inFlight) || !String.Equals(inFlight.Message.Id, entry.Id, StringComparison.Ordinal)) + throw new ReceiptExpiredException(); + + Interlocked.Increment(ref state.Abandoned); + var redelivered = inFlight.Message with { DeliveryCount = entry.DeliveryCount + 1 }; + ScheduleRedelivery(receipt.Destination, redelivered, redeliveryDelay); + + return Task.CompletedTask; + } + + public Task RenewLockAsync(TransportEntry entry, TimeSpan? duration, CancellationToken ct) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(entry); + + var receipt = GetReceipt(entry); + var state = GetExistingDestination(receipt.Destination); + + if (!state.InFlight.TryGetValue(receipt.LockToken, out var inFlight) || !String.Equals(inFlight.Message.Id, entry.Id, StringComparison.Ordinal)) + throw new ReceiptExpiredException(); + + // Renewal only extends a finite visibility window. A message received without a window holds an indefinite + // lock, so there is nothing to extend — leave it as-is rather than imposing a window that could reclaim it. + if (inFlight.VisibilityExpiresUtc is null) + return Task.CompletedTask; + + var renewed = inFlight with { VisibilityExpiresUtc = _timeProvider.GetUtcNow().Add(duration ?? _defaultLockRenewal) }; + if (!state.InFlight.TryUpdate(receipt.LockToken, renewed, inFlight)) + throw new ReceiptExpiredException(); + + // Re-arm the reclaim wake for the extended window. + ScheduleReclaim(state, duration ?? _defaultLockRenewal); + return Task.CompletedTask; + } + + public Task DeadLetterAsync(TransportEntry entry, string? reason, CancellationToken ct) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(entry); + + var receipt = GetReceipt(entry); + var state = GetExistingDestination(receipt.Destination); + + if (!state.InFlight.TryRemove(receipt.LockToken, out var inFlight) || !String.Equals(inFlight.Message.Id, entry.Id, StringComparison.Ordinal)) + throw new ReceiptExpiredException(); + + // Dead-letter with the caller's entry headers (which may carry forensics stamped by the core), not the + // originally-stored ones. + DeadLetter(state, inFlight.Message with { Headers = entry.Headers }, reason); + return Task.CompletedTask; + } + + public Task> ReceiveDeadLetteredAsync(DestinationAddress destination, ReceiveRequest request, CancellationToken ct) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(destination); + ArgumentNullException.ThrowIfNull(request); + + if (!_destinations.TryGetValue(ReceivableKey(destination), out var state)) + return Task.FromResult>([]); + + int maxMessages = request.MaxMessages <= 0 ? 1 : request.MaxMessages; + var entries = new List(maxMessages); + while (entries.Count < maxMessages && state.TryReadDeadletter(out var message)) + { + entries.Add(new TransportEntry + { + Id = message.Id, + Destination = destination, + Body = message.Body, + Headers = message.Headers, + DeliveryCount = message.DeliveryCount, + EnqueuedUtc = message.EnqueuedUtc, + Receipt = new Receipt { TransportState = null } + }); + } + + return Task.FromResult>(entries); + } + + public Task SubscribeAsync(DestinationAddress source, Func onMessage, PushOptions options, CancellationToken ct) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(source); + ArgumentNullException.ThrowIfNull(onMessage); + ArgumentNullException.ThrowIfNull(options); + + var subscription = new PushSubscription(source); + subscription.Start(RunPushSubscriptionAsync(source, onMessage, options, subscription.CancellationToken)); + return Task.FromResult(subscription); + } + + public Task GetStatsAsync(DestinationAddress destination, CancellationToken ct) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(destination); + + if (!_destinations.TryGetValue(ReceivableKey(destination), out var state)) + return Task.FromResult(new MessageDestinationStats()); + + return Task.FromResult(new MessageDestinationStats + { + Queued = state.QueuedCount, + Working = state.InFlight.Count, + Deadletter = state.DeadletterCount, + Enqueued = Volatile.Read(ref state.Enqueued), + Dequeued = Volatile.Read(ref state.Dequeued), + Completed = Volatile.Read(ref state.Completed), + Abandoned = Volatile.Read(ref state.Abandoned) + }); + } + + public Task EnsureAsync(IReadOnlyList declarations, CancellationToken ct) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(declarations); + + foreach (var declaration in declarations) + { + var address = declaration.Address; + ArgumentNullException.ThrowIfNull(address); + + switch (address.Role) + { + case DestinationRole.Queue: + GetOrAddDestination(StorageKey(address)); + break; + case DestinationRole.Topic: + _roles.TryAdd(StorageKey(address), DestinationRole.Topic); + _topicSubscriptions.GetOrAdd(StorageKey(address), static _ => new ConcurrentDictionary(StringComparer.OrdinalIgnoreCase)); + break; + case DestinationRole.Subscription: + if (String.IsNullOrEmpty(address.Topic)) + throw new ArgumentException("A subscription declaration must specify its owning topic.", nameof(declarations)); + + AddTopicSubscription(address.Topic, StorageKey(address)); + break; + case DestinationRole.Binding: + if (String.IsNullOrEmpty(address.Topic)) + throw new ArgumentException("A binding declaration must specify a source topic.", nameof(declarations)); + + AddTopicSubscription(address.Topic, StorageKey(address)); + break; + default: + throw new ArgumentOutOfRangeException(nameof(declarations), address.Role, "Unsupported destination role."); + } + } + + return Task.CompletedTask; + } + + public Task DeleteAsync(DestinationAddress destination, CancellationToken ct) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(destination); + + string key = StorageKey(destination); + _roles.TryRemove(key, out _); + if (_destinations.TryRemove(key, out var removed)) + removed.Complete(); + _topicSubscriptions.TryRemove(key, out _); + + foreach (var subscriptions in _topicSubscriptions.Values) + subscriptions.TryRemove(key, out _); + + return Task.CompletedTask; + } + + public Task ExistsAsync(DestinationAddress destination, CancellationToken ct) + { + ThrowIfDisposed(); + ct.ThrowIfCancellationRequested(); + ArgumentNullException.ThrowIfNull(destination); + + return Task.FromResult(_roles.ContainsKey(StorageKey(destination))); + } + + public ValueTask DisposeAsync() + { + if (Interlocked.Exchange(ref _isDisposed, 1) == 1) + return ValueTask.CompletedTask; + + _disposeCancellationTokenSource.Cancel(); + _disposeCancellationTokenSource.Dispose(); + + foreach (var timer in _redeliveryTimers.Keys) + { + if (_redeliveryTimers.TryRemove(timer, out _)) + timer.Dispose(); + } + + _destinations.Clear(); + _roles.Clear(); + _topicSubscriptions.Clear(); + return ValueTask.CompletedTask; + } + + private async Task RunPushSubscriptionAsync(DestinationAddress source, Func onMessage, PushOptions options, CancellationToken subscriptionCancellationToken) + { + using var linkedCancellationTokenSource = CancellationTokenSource.CreateLinkedTokenSource(subscriptionCancellationToken, _disposeCancellationTokenSource.Token); + var token = linkedCancellationTokenSource.Token; + int maxMessages = Math.Max(1, options.MaxConcurrentMessages); + + while (!token.IsCancellationRequested) + { + IReadOnlyList entries; + try + { + entries = await ReceiveAsync(source, new ReceiveRequest + { + MaxMessages = maxMessages, + MaxWaitTime = options.PollInterval + }, token).AnyContext(); + } + catch (OperationCanceledException) when (token.IsCancellationRequested) + { + break; + } + + foreach (var entry in entries) + { + try + { + await onMessage(entry, token).AnyContext(); + } + catch (OperationCanceledException) when (token.IsCancellationRequested) + { + break; + } + catch + { + // Safety net: the handler threw without settling, so abandon for redelivery. If the handler had + // already settled the message (e.g. dead-lettered a poison payload and then rethrew), the receipt + // is gone — treat that as already handled rather than faulting the subscription loop. + try + { + await AbandonAsync(entry, token).AnyContext(); + } + catch (ReceiptExpiredException) + { + } + catch (OperationCanceledException) when (token.IsCancellationRequested) + { + break; + } + } + } + } + } + + private void EnqueueForDestination(string key, StoredMessage message) + { + // Topic sends fan out one copy per subscription; a topic with no subscriptions drops the message (real + // pub/sub semantics — subscriptions must exist before a publish can reach them). + if (key.StartsWith("t:", StringComparison.Ordinal)) + { + if (!_topicSubscriptions.TryGetValue(key, out var subscriptions)) + return; + + foreach (string subscription in subscriptions.Keys) + EnqueueStoredMessage(subscription, message with { Destination = subscription }); + + return; + } + + EnqueueStoredMessage(key, message); + } + + private void EnqueueStoredMessage(string key, StoredMessage message) + { + var state = GetOrAddDestination(key); + state.Enqueue(message with { Destination = key }); + } + + // Make an abandoned message invisible for the redelivery delay, then re-enqueue it. Re-enqueueing releases the + // destination's availability semaphore, so a consumer blocked in a long receive wait wakes immediately when the + // message becomes due. The one-shot timer is tracked so it can be disposed if the transport is torn down first. + private void ScheduleRedelivery(string destination, StoredMessage message, TimeSpan delay) + { + ITimer? timer = null; + timer = _timeProvider.CreateTimer(timerState => + { + if (timer is not null && _redeliveryTimers.TryRemove(timer, out _)) + timer.Dispose(); + + if (Volatile.Read(ref _isDisposed) == 1) + return; + + try + { + EnqueueStoredMessage(destination, message); + } + catch (ObjectDisposedException) { } + catch (InvalidOperationException) { } // destination was deleted / completed between scheduling and firing + }, null, delay, Timeout.InfiniteTimeSpan); + + _redeliveryTimers[timer] = 0; + + // A redelivery scheduled right as the transport disposes could otherwise leak its timer; clean up the race. + if (Volatile.Read(ref _isDisposed) == 1 && _redeliveryTimers.TryRemove(timer, out _)) + timer.Dispose(); + } + + // Fires shortly after a visibility window lapses and reclaims any expired in-flight messages, which re-enqueues + // them and releases the destination's availability semaphore — waking a consumer blocked in a long receive. + // ReclaimExpired re-checks each message's current expiry, so a renewed or already-settled message is left alone. + private void ScheduleReclaim(DestinationState state, TimeSpan delay) + { + // Small buffer so the timer fires just after expiry rather than racing it (clock granularity). + var fireAfter = delay + TimeSpan.FromMilliseconds(50); + + ITimer? timer = null; + timer = _timeProvider.CreateTimer(timerState => + { + if (timer is not null && _redeliveryTimers.TryRemove(timer, out _)) + timer.Dispose(); + + if (Volatile.Read(ref _isDisposed) == 1) + return; + + try + { + state.ReclaimExpired(_timeProvider.GetUtcNow()); + } + catch (ObjectDisposedException) { } + catch (InvalidOperationException) { } // destination was completed/deleted between scheduling and firing + }, null, fireAfter, Timeout.InfiniteTimeSpan); + + _redeliveryTimers[timer] = 0; + + if (Volatile.Read(ref _isDisposed) == 1 && _redeliveryTimers.TryRemove(timer, out _)) + timer.Dispose(); + } + + private bool TryReceive(DestinationAddress source, DestinationState state, TimeSpan? visibility, out TransportEntry entry) + { + while (state.TryDequeue(out var message)) + { + if (IsExpired(message)) + { + DeadLetter(state, message, "expired"); + continue; + } + + // The receipt carries the internal (role-qualified) key so settlement resolves the same state; the entry's + // Destination stays the caller-facing source address. + var receipt = new InMemoryReceipt(ReceivableKey(source), Guid.NewGuid().ToString("N")); + DateTimeOffset? visibilityExpiresUtc = visibility is { } window ? _timeProvider.GetUtcNow().Add(window) : null; + state.InFlight[receipt.LockToken] = new InFlightMessage(message, receipt, visibilityExpiresUtc); + Interlocked.Increment(ref state.Dequeued); + + // Schedule a reclaim at the visibility expiry so a consumer blocked in a long receive wakes when the lease + // lapses (matching real brokers like SQS), rather than only being reclaimed at the next receive call. + if (visibility is { } visibilityWindow) + ScheduleReclaim(state, visibilityWindow); + + entry = new TransportEntry + { + Id = message.Id, + Destination = source, + Body = message.Body, + Headers = message.Headers, + DeliveryCount = message.DeliveryCount, + EnqueuedUtc = message.EnqueuedUtc, + Receipt = new Receipt { TransportState = receipt } + }; + return true; + } + + entry = null!; + return false; + } + + private bool IsExpired(StoredMessage message) + { + string? expiration = message.Headers.GetValueOrDefault(KnownHeaders.Expiration); + if (expiration is null) + return false; + + return DateTimeOffset.TryParse(expiration, CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal, out var expiresAt) + && expiresAt <= _timeProvider.GetUtcNow(); + } + + private void DeadLetter(DestinationState state, StoredMessage message, string? reason) + { + if (!String.IsNullOrEmpty(reason)) + message = message with { Headers = message.Headers.ToBuilder().Set(KnownHeaders.DeadLetterReason, reason).Build() }; + + state.Deadletter(message); + } + + private StoredMessage CreateStoredMessage(string destination, string messageId, TransportMessage message, TransportSendOptions options) + { + var headers = message.Headers.ToBuilder() + .SetIfMissing(KnownHeaders.Priority, options.Priority.ToString()) + .Build(); + int deliveryCount = Int32.TryParse(headers.GetValueOrDefault(KnownHeaders.Attempts), NumberStyles.Integer, CultureInfo.InvariantCulture, out int attempts) && attempts > 0 + ? attempts + : 1; + + return new StoredMessage( + messageId, + destination, + message.Body.ToArray(), + headers, + NormalizePriority(options.Priority), + DeliveryCount: deliveryCount, + EnqueuedUtc: _timeProvider.GetUtcNow()); + } + + // Internal state is keyed by role-qualified names derived from the canonical address: "t:" for topics, "q:" for + // every receivable destination (queues AND subscriptions — a subscription is a queue-shaped destination a topic + // fans into, exactly like an SNS-bound SQS queue, keyed by its topic-qualified address key). This gives a + // queue/subscription and a topic sharing a route name distinct namespaces, as real brokers do. + private static string StorageKey(DestinationAddress address) => + address.Role == DestinationRole.Topic ? "t:" + address.Name : "q:" + address.Key; + + // Receive-path keys are always queue-shaped; receive/stats/dead-letter reads never target a topic. + private static string ReceivableKey(DestinationAddress address) => "q:" + address.Key; + + private static DestinationRole RoleForKey(string key) => key[0] == 't' ? DestinationRole.Topic : DestinationRole.Queue; + + private DestinationState GetOrAddDestination(string key) + { + _roles.TryAdd(key, RoleForKey(key)); + return _destinations.GetOrAdd(key, static _ => new DestinationState()); + } + + private DestinationState GetExistingDestination(string key) + { + if (_destinations.TryGetValue(key, out var destination)) + return destination; + + throw new ReceiptExpiredException($"The destination \"{key}\" no longer exists."); + } + + private void AddTopicSubscription(string topic, string subscriptionStorageKey) + { + string topicKey = "t:" + topic; + _roles.TryAdd(topicKey, DestinationRole.Topic); + GetOrAddDestination(subscriptionStorageKey); + var subscriptions = _topicSubscriptions.GetOrAdd(topicKey, static _ => new ConcurrentDictionary(StringComparer.OrdinalIgnoreCase)); + subscriptions[subscriptionStorageKey] = 0; + } + + private static MessagePriority NormalizePriority(MessagePriority priority) + { + return priority switch + { + MessagePriority.Low => MessagePriority.Low, + MessagePriority.Normal => MessagePriority.Normal, + MessagePriority.High => MessagePriority.High, + _ => MessagePriority.Normal + }; + } + + private static InMemoryReceipt GetReceipt(TransportEntry entry) + { + return entry.Receipt.TransportState as InMemoryReceipt ?? throw new ReceiptExpiredException(); + } + + private void ThrowIfDisposed() + { + ObjectDisposedException.ThrowIf(Volatile.Read(ref _isDisposed) == 1, this); + } + + private sealed record StoredMessage( + string Id, + string Destination, + ReadOnlyMemory Body, + MessageHeaders Headers, + MessagePriority Priority, + int DeliveryCount, + DateTimeOffset EnqueuedUtc); + + private sealed record InFlightMessage(StoredMessage Message, InMemoryReceipt Receipt, DateTimeOffset? VisibilityExpiresUtc); + + private sealed record InMemoryReceipt(string Destination, string LockToken); + + private sealed class DestinationState + { + private readonly Channel[] _channels = + [ + Channel.CreateUnbounded(CreateChannelOptions()), + Channel.CreateUnbounded(CreateChannelOptions()), + Channel.CreateUnbounded(CreateChannelOptions()) + ]; + + private readonly Channel _deadletterChannel = Channel.CreateUnbounded(CreateChannelOptions()); + private readonly SemaphoreSlim _availableMessages = new(0); + private long _queuedCount; + private long _deadletterCount; + private int _isCompleted; + + public ConcurrentDictionary InFlight { get; } = new(StringComparer.Ordinal); + public long Enqueued; + public long Dequeued; + public long Completed; + public long Abandoned; + public long Deadlettered; + + public long QueuedCount => Volatile.Read(ref _queuedCount); + public long DeadletterCount => Volatile.Read(ref _deadletterCount); + + public void Enqueue(StoredMessage message) + { + if (!_channels[(int)message.Priority].Writer.TryWrite(message)) + throw new InvalidOperationException("The destination is no longer accepting messages."); + + Interlocked.Increment(ref _queuedCount); + Interlocked.Increment(ref Enqueued); + _availableMessages.Release(); + } + + public bool TryDequeue(out StoredMessage message) + { + for (int index = (int)MessagePriority.High; index >= (int)MessagePriority.Low; index--) + { + if (_channels[index].Reader.TryRead(out message!)) + { + _availableMessages.Wait(0); + Interlocked.Decrement(ref _queuedCount); + return true; + } + } + + message = null!; + return false; + } + + public async ValueTask WaitToReadAsync(CancellationToken cancellationToken) + { + if (QueuedCount > 0) + return true; + + await _availableMessages.WaitAsync(cancellationToken).ConfigureAwait(false); + return true; + } + + public void Deadletter(StoredMessage message) + { + if (!_deadletterChannel.Writer.TryWrite(message)) + throw new InvalidOperationException("The destination is no longer accepting dead-letter messages."); + + Interlocked.Increment(ref _deadletterCount); + Interlocked.Increment(ref Deadlettered); + } + + public bool TryReadDeadletter(out StoredMessage message) + { + if (_deadletterChannel.Reader.TryRead(out message!)) + { + Interlocked.Decrement(ref _deadletterCount); + return true; + } + + message = null!; + return false; + } + + public void ReclaimExpired(DateTimeOffset now) + { + if (Volatile.Read(ref _isCompleted) == 1) + return; + + foreach (var kvp in InFlight) + { + if (kvp.Value.VisibilityExpiresUtc is { } expiry && expiry <= now && InFlight.TryRemove(kvp.Key, out var inFlight)) + { + Interlocked.Increment(ref Abandoned); + Enqueue(inFlight.Message with { DeliveryCount = inFlight.Message.DeliveryCount + 1 }); + } + } + } + + public void Complete() + { + if (Interlocked.Exchange(ref _isCompleted, 1) == 1) + return; + + foreach (var channel in _channels) + channel.Writer.TryComplete(); + + _deadletterChannel.Writer.TryComplete(); + } + + private static UnboundedChannelOptions CreateChannelOptions() + { + return new UnboundedChannelOptions + { + AllowSynchronousContinuations = false, + SingleReader = false, + SingleWriter = false + }; + } + } + + private sealed class PushSubscription : IPushSubscription + { + private readonly CancellationTokenSource _cancellationTokenSource = new(); + private Task? _worker; + + public PushSubscription(DestinationAddress source) + { + Source = source; + } + + public DestinationAddress Source { get; } + public CancellationToken CancellationToken => _cancellationTokenSource.Token; + + public void Start(Task worker) + { + _worker = worker; + } + + public async ValueTask DisposeAsync() + { + await _cancellationTokenSource.CancelAsync().AnyContext(); + + if (_worker is not null) + { + try + { + await _worker.AnyContext(); + } + catch (OperationCanceledException) { } + } + + _cancellationTokenSource.Dispose(); + } + } +} diff --git a/src/Foundatio/Messaging/KnownHeaders.cs b/src/Foundatio/Messaging/KnownHeaders.cs new file mode 100644 index 000000000..f8dd99f0d --- /dev/null +++ b/src/Foundatio/Messaging/KnownHeaders.cs @@ -0,0 +1,23 @@ +namespace Foundatio.Messaging; + +public static class KnownHeaders +{ + public const string MessageType = "message.type"; + public const string ContentType = "message.content_type"; + public const string CorrelationId = "message.correlation_id"; + public const string TraceParent = "traceparent"; + public const string TraceState = "tracestate"; + public const string Priority = "message.priority"; + public const string Expiration = "message.expiration"; + public const string Attempts = "message.attempts"; + public const string DeadLetterReason = "message.dead_letter.reason"; + + // Forensics stamped by the core when a message is dead-lettered, so a dead message is triageable with plain + // transport tooling. These names are a compatibility contract; values are truncated to fit transport limits. + public const string DeadLetterAttempts = "message.dead_letter.attempts"; + public const string DeadLetterExceptionType = "message.dead_letter.exception_type"; + public const string DeadLetterExceptionMessage = "message.dead_letter.exception_message"; + public const string DeadLetterExceptionStackTrace = "message.dead_letter.exception_stack"; + public const string DeadLetterFailedAt = "message.dead_letter.failed_at"; + public const string DeadLetterOriginalDestination = "message.dead_letter.original_destination"; +} diff --git a/src/Foundatio/Messaging/LegacyMessageBusException.cs b/src/Foundatio/Messaging/LegacyMessageBusException.cs new file mode 100644 index 000000000..aef314810 --- /dev/null +++ b/src/Foundatio/Messaging/LegacyMessageBusException.cs @@ -0,0 +1,17 @@ +using System; + +namespace Foundatio.Messaging.Legacy; + +/// +/// Exception thrown when a legacy message bus operation fails. +/// +public class MessageBusException : Exception +{ + public MessageBusException(string message) : base(message) + { + } + + public MessageBusException(string message, Exception innerException) : base(message, innerException) + { + } +} diff --git a/src/Foundatio/Messaging/Message.cs b/src/Foundatio/Messaging/Message.cs index eeaad8fc5..e943c9962 100644 --- a/src/Foundatio/Messaging/Message.cs +++ b/src/Foundatio/Messaging/Message.cs @@ -3,7 +3,7 @@ using System.Diagnostics; using System.Diagnostics.CodeAnalysis; -namespace Foundatio.Messaging; +namespace Foundatio.Messaging.Legacy; /// /// Represents a message received from the message bus with metadata and raw payload. diff --git a/src/Foundatio/Messaging/MessageBus.cs b/src/Foundatio/Messaging/MessageBus.cs new file mode 100644 index 000000000..87089ea2a --- /dev/null +++ b/src/Foundatio/Messaging/MessageBus.cs @@ -0,0 +1,548 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Serializer; +using Foundatio.Utility; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; + +namespace Foundatio.Messaging; + +public sealed record MessageSendOptions +{ + public MessagePriority Priority { get; init; } = MessagePriority.Normal; + public TimeSpan? Delay { get; init; } + public DateTimeOffset? DeliverAt { get; init; } + public TimeSpan? TimeToLive { get; init; } + public string? CorrelationId { get; init; } + /// Overrides the routed destination for this send. + public string? Destination { get; init; } + public MessageHeaders? Headers { get; init; } +} + +public sealed record MessagePublishOptions +{ + public MessagePriority Priority { get; init; } = MessagePriority.Normal; + public TimeSpan? Delay { get; init; } + public DateTimeOffset? DeliverAt { get; init; } + public TimeSpan? TimeToLive { get; init; } + public string? CorrelationId { get; init; } + /// Overrides the routed topic for this publish. + public string? Topic { get; init; } + public MessageHeaders? Headers { get; init; } +} + +/// +/// The delivery channels a subscription consumes. By default a handler listens on both of its type's channels; a +/// handler that only ever consumes commands (or only events) states that intent so no idle listener is wired — and so +/// a queue-only or topic-only transport can serve it. +/// +[Flags] +public enum MessageDeliveries +{ + /// Sent commands from the type's queue-role destination (competing consumers). + Sent = 1, + + /// Published events, delivered via this subscriber's group on the type's topic. + Published = 2, + + Both = Sent | Published +} + +/// +/// Options for attaching a handler to a message type — via AddFoundatio().Messaging.AddHandler<T, THandler>(o => ...) +/// or programmatically via . By default a subscription listens on the +/// type's two delivery channels — sent messages (one handler instance across the fleet processes each) and published +/// messages (delivered per the subscription identity below) — narrowed by . +/// +public sealed class MessageSubscriptionOptions +{ + /// + /// Which delivery channels this subscription consumes. Default : on a + /// transport that supports only one channel's roles, the unsupported channel is skipped (logged at debug). + /// Explicitly requesting a single channel the transport cannot serve throws . + /// + public MessageDeliveries Deliveries { get; set; } = MessageDeliveries.Both; + + /// + /// When true, published messages are received by EVERY running instance (each takes a unique subscription), + /// instead of once per service. For per-instance local state — cache invalidation, config reload. Mutually + /// exclusive with . Does not affect sent messages, which always go to exactly one instance. + /// + public bool PerInstance { get; set; } + + /// + /// The subscriber-group identity for published messages. Defaults to the service identity (plus the + /// when set), so all instances of a service share one subscription and compete + /// (each published message is handled once per service). Set an explicit name to form an independent named + /// subscriber group. + /// + public string? Subscription { get; set; } + + /// + /// Distinguishes this subscriber group from others in the same service when no explicit + /// is set — the default group becomes "{service-identity}.{qualifier}". Set automatically to the handler type name + /// by AddHandler<T, THandler> so each handler class receives its own copy of published messages. + /// Ignored when or is set. + /// + public string? SubscriptionQualifier { get; set; } + + /// + /// Maximum messages this subscription processes concurrently per instance. Default 1 — a deliberate divergence + /// from libraries that default higher: 1 is the only default that preserves per-handler ordering, each handler + /// already gets its own concurrent stream (10 handlers = 10 parallel consumers), and scaling out replicas scales + /// throughput without giving up ordering per instance. Raise it for handlers that are I/O-bound and order-agnostic. + /// + public int MaxConcurrency { get; set; } = 1; + + /// Maximum delivery attempts before dead-lettering. Null uses the default . + public int? MaxAttempts { get; set; } + + /// Delay before each redelivery given the 1-based attempt number. Null uses the default . + public Func? RedeliveryBackoff { get; set; } + + /// + /// Marks a handler failure as unrecoverable: when the predicate returns true the message is dead-lettered + /// immediately instead of retried. Null uses the default . Prefer + /// for the common by-type case. + /// + public Func? DeadLetterWhen { get; set; } + + /// Whether messages auto-complete when the handler returns (default) or are settled manually. + public AckMode AckMode { get; set; } = AckMode.Auto; + + /// Routes by a different type than the handler's type parameter (grouped/interface consumers). + public Type? RouteType { get; set; } + + /// Overrides the routed send destination this subscription listens on. + public string? Destination { get; set; } + + /// Overrides the routed topic this subscription listens on. + public string? Topic { get; set; } + + /// + /// Consumer identity. Subscriptions sharing a key on the same channel form one consumer group and compete; + /// defaults to a per-channel key derived from the route. Subscriptions sharing a key must configure identical + /// failure policies — the backoff/DeadLetterWhen DELEGATES are compared by identity, so share the same delegate + /// instances (a lambda recreated per subscription will be rejected as a conflicting registration). + /// + public string? Key { get; set; } + + /// + /// Dead-letters failures of type immediately instead of retrying — for + /// exceptions a retry can never fix (validation, malformed data). Composes: call once per exception type. + /// + public MessageSubscriptionOptions DeadLetterOn() where TException : Exception + { + var existing = DeadLetterWhen; + DeadLetterWhen = existing is null + ? static ex => ex is TException + : ex => existing(ex) || ex is TException; + return this; + } +} + +/// +/// A started subscription; disposing detaches the handler from the message type's delivery channels. Channel-specific +/// properties are empty when that channel was not wired (see ). +/// +public interface IMessageSubscription : IAsyncDisposable +{ + /// Consumer identity; subscriptions sharing a key on a channel form one competing group. + string Key { get; } + + /// The send-channel destination this subscription listens on. + string Destination { get; } + + /// The publish-channel topic this subscription listens on. + string Topic { get; } + + /// The publish-channel subscriber-group identity (service identity unless overridden or per-instance). + string Subscription { get; } + + /// + /// The publish-channel transport source: the topic-qualified subscription address, so the same subscription + /// identity on two topics resolves to two distinct sources. + /// + string Source { get; } +} + +/// +/// The messaging client. Handlers are registered without any topology decision and the caller's verb carries the +/// delivery semantic: +/// +/// — a command / unit of work: exactly one handler instance across the fleet +/// processes it (competing consumers). +/// — an event: every subscribing service receives one copy (a scaled service's +/// instances compete for it), or every instance when the subscription opts into +/// . +/// +/// Retry and dead-lettering are core-owned and identical for both verbs: a handler that throws triggers redelivery +/// and, once attempts are exhausted, the dead-letter policy. +/// +public interface IMessageBus : IAsyncDisposable +{ + /// Sends a command / unit of work; exactly one handler instance across the fleet processes it. + Task SendAsync(T message, MessageSendOptions? options = null, CancellationToken cancellationToken = default) where T : class; + Task SendBatchAsync(IEnumerable messages, MessageSendOptions? options = null, CancellationToken cancellationToken = default) where T : class; + Task SendBatchAsync(IEnumerable messages, MessageSendOptions? options = null, CancellationToken cancellationToken = default); + + /// Publishes an event; each subscribing service receives one copy (its instances compete). + Task PublishAsync(T message, MessagePublishOptions? options = null, CancellationToken cancellationToken = default) where T : class; + Task PublishBatchAsync(IEnumerable messages, MessagePublishOptions? options = null, CancellationToken cancellationToken = default) where T : class; + Task PublishBatchAsync(IEnumerable messages, MessagePublishOptions? options = null, CancellationToken cancellationToken = default); + + /// + /// Attaches a handler to the message type's delivery channels (sent and published messages). Prefer declarative + /// registration (AddFoundatio().Messaging.AddHandler<T, THandler>()) for handlers that live for the + /// app's lifetime; use this for dynamic subscriptions. + /// + Task SubscribeAsync(Func, CancellationToken, Task> handler, MessageSubscriptionOptions? options = null, CancellationToken cancellationToken = default) where T : class; + Task SubscribeAsync(Func handler, MessageSubscriptionOptions? options = null, CancellationToken cancellationToken = default); +} + +/// +/// Governs the messaging client's topology-administration behavior. Publishing and subscribing never implicitly grant +/// themselves more than this mode allows, so an app on a locked-down broker can state "validate only" or "never touch +/// topology" instead of hoping implicit creation fails gracefully. +/// +/// +/// The mode governs the CORE's provisioning calls. A transport may still lazily create cheap local structures on its +/// own paths (e.g. consumer groups on first receive); combine Validate/None with the transport's own knobs (such as +/// AwsMessageTransportOptions.AutoCreateDestinations = false) for a fully locked-down broker. +/// +public enum TopologyMode +{ + /// Create missing destinations on first use and at handler-host startup (default). + Ensure, + + /// Never create. Verify each destination exists on first use (cached) and throw when missing. + Validate, + + /// No topology calls at all; destinations are assumed pre-provisioned out of band. + None +} + +public sealed record MessageBusOptions +{ + /// How the client administers topology (create on use, validate-only, or never touch). Default . + public TopologyMode Topology { get; init; } = TopologyMode.Ensure; + + public ISerializer Serializer { get; init; } = DefaultSerializer.Instance; + public string ContentType { get; init; } = "application/json"; + public IMessageRouter Router { get; init; } = DefaultMessageRouter.Instance; + public IMessageTypeRegistry MessageTypes { get; init; } = new MessageTypeRegistry(); + /// + /// Enables durable scheduling: delayed sends beyond a transport ceiling and store-parked retry delays are written + /// here and drained by the job runtime pump. Messaging depends only on the dispatch-storage contract — any + /// satisfies it, but a provider can implement + /// alone. The DI builder registers the pump automatically with the store; when wiring options by hand, ensure a + /// pump (JobRuntimePumpService / JobScheduleProcessor) is running or parked messages will never be dispatched. + /// + public IScheduledDispatchStore? RuntimeStore { get; init; } + public RetryPolicy RetryPolicy { get; init; } = new(); + + /// + /// Whether disposing this bus also disposes the transport. True (default) for a transport the bus solely uses; set + /// false when the transport is a shared/externally-owned instance (e.g. a DI singleton). + /// + public bool OwnsTransport { get; init; } = true; + public TimeProvider TimeProvider { get; init; } = TimeProvider.System; + public ILoggerFactory? LoggerFactory { get; init; } +} + +/// +/// The one messaging client over the transport. Routing, serialization, settlement, scheduling, and the consumer loop +/// live in ; this type maps the two delivery verbs and subscriptions onto that core. +/// +public sealed class MessageBus : IMessageBus +{ + private readonly MessageClientCore _core; + private readonly ILogger _logger; + + public MessageBus(IMessageTransport transport, MessageBusOptions? options = null) + { + ArgumentNullException.ThrowIfNull(transport); + options ??= new MessageBusOptions(); + _logger = (options.LoggerFactory ?? NullLoggerFactory.Instance).CreateLogger(); + _core = new MessageClientCore(transport, options.Serializer, options.Router, options.RuntimeStore, options.TimeProvider, _logger, + static (message, inner) => inner is null ? new MessageBusException(message) : new MessageBusException(message, inner), options.RetryPolicy, options.OwnsTransport, options.MessageTypes, options.ContentType, options.Topology); + } + + public Task SendAsync(T message, MessageSendOptions? options = null, CancellationToken cancellationToken = default) where T : class + { + ArgumentNullException.ThrowIfNull(message); + options ??= new MessageSendOptions(); + return _core.SendAsync(ScheduledDispatchKind.QueueMessage, typeof(T), message, ToEnvelope(options), GetDestination(typeof(T), options.Destination), ensureDestination: null, cancellationToken); + } + + public Task SendBatchAsync(IEnumerable messages, MessageSendOptions? options = null, CancellationToken cancellationToken = default) where T : class + { + ArgumentNullException.ThrowIfNull(messages); + options ??= new MessageSendOptions(); + return _core.SendBatchAsync(ScheduledDispatchKind.QueueMessage, messages.Cast(), typeof(T), ToEnvelope(options), type => GetDestination(type, options.Destination), ensureDestination: null, cancellationToken); + } + + public Task SendBatchAsync(IEnumerable messages, MessageSendOptions? options = null, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(messages); + options ??= new MessageSendOptions(); + return _core.SendBatchAsync(ScheduledDispatchKind.QueueMessage, messages, null, ToEnvelope(options), type => GetDestination(type, options.Destination), ensureDestination: null, cancellationToken); + } + + public Task PublishAsync(T message, MessagePublishOptions? options = null, CancellationToken cancellationToken = default) where T : class + { + ArgumentNullException.ThrowIfNull(message); + options ??= new MessagePublishOptions(); + return _core.SendAsync(ScheduledDispatchKind.PubSubMessage, typeof(T), message, ToEnvelope(options), GetTopic(typeof(T), options.Topic), EnsureTopicAsync, cancellationToken); + } + + public Task PublishBatchAsync(IEnumerable messages, MessagePublishOptions? options = null, CancellationToken cancellationToken = default) where T : class + { + ArgumentNullException.ThrowIfNull(messages); + options ??= new MessagePublishOptions(); + return _core.SendBatchAsync(ScheduledDispatchKind.PubSubMessage, messages.Cast(), typeof(T), ToEnvelope(options), type => GetTopic(type, options.Topic), EnsureTopicAsync, cancellationToken); + } + + public Task PublishBatchAsync(IEnumerable messages, MessagePublishOptions? options = null, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(messages); + options ??= new MessagePublishOptions(); + return _core.SendBatchAsync(ScheduledDispatchKind.PubSubMessage, messages, null, ToEnvelope(options), type => GetTopic(type, options.Topic), EnsureTopicAsync, cancellationToken); + } + + public Task SubscribeAsync(Func, CancellationToken, Task> handler, MessageSubscriptionOptions? options = null, CancellationToken cancellationToken = default) where T : class + { + ArgumentNullException.ThrowIfNull(handler); + return SubscribeCoreAsync(options, typeof(T), (config, token) => _core.StartListenerAsync(config, handler, token), cancellationToken); + } + + public Task SubscribeAsync(Func handler, MessageSubscriptionOptions? options = null, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(handler); + return SubscribeCoreAsync(options, typeof(object), (config, token) => _core.StartListenerAsync(config, handler, token), cancellationToken); + } + + private async Task SubscribeCoreAsync(MessageSubscriptionOptions? options, Type fallbackType, Func> start, CancellationToken cancellationToken) + { + var deliveries = options?.Deliveries ?? MessageDeliveries.Both; + if ((deliveries & MessageDeliveries.Both) == 0) + throw new ArgumentException("Deliveries must include at least one delivery channel.", nameof(options)); + + bool wantSent = deliveries.HasFlag(MessageDeliveries.Sent); + bool wantPublished = deliveries.HasFlag(MessageDeliveries.Published); + bool canSend = _core.SupportsRole(DestinationRole.Queue); + bool canPublish = _core.SupportsRole(DestinationRole.Topic) && _core.SupportsRole(DestinationRole.Subscription); + + // An explicit single-channel request the transport cannot serve is a configuration error and must fail loudly; + // the default Both narrows to whatever the transport supports (a queue-only transport still serves commands) + // but a transport that can serve neither channel is always an error. + if (deliveries != MessageDeliveries.Both) + { + if (wantSent && !canSend) + throw new NotSupportedException($"Subscription requests {nameof(MessageDeliveries.Sent)} deliveries, but the transport does not support {DestinationRole.Queue} destinations."); + if (wantPublished && !canPublish) + throw new NotSupportedException($"Subscription requests {nameof(MessageDeliveries.Published)} deliveries, but the transport does not support {DestinationRole.Topic} and {DestinationRole.Subscription} destinations."); + } + else if (!canSend && !canPublish) + { + throw new NotSupportedException("The transport supports neither queue nor topic/subscription destinations; no delivery channel can be wired."); + } + + bool wireSent = wantSent && canSend; + bool wirePublished = wantPublished && canPublish; + + if (wantSent && !wireSent) + _logger.LogDebug("Skipping the sent-message channel for {MessageType}: the transport does not support {Role} destinations", fallbackType.Name, DestinationRole.Queue); + if (wantPublished && !wirePublished) + _logger.LogDebug("Skipping the published-message channel for {MessageType}: the transport does not support {TopicRole}/{SubscriptionRole} destinations", fallbackType.Name, DestinationRole.Topic, DestinationRole.Subscription); + + var channels = BuildChannels(options, fallbackType); + MessageListenerHandle? sent = wireSent ? await start(channels.Send, cancellationToken).AnyContext() : null; + try + { + MessageListenerHandle? published = null; + if (wirePublished) + { + await EnsureSubscriptionAsync(channels.Publish, cancellationToken).AnyContext(); + published = await start(channels.Publish, cancellationToken).AnyContext(); + } + + LogSubscription(channels.Send, channels.Publish, wireSent, wirePublished); + return new MessageSubscription(sent, published); + } + catch + { + if (sent is not null) + await sent.DisposeAsync().AnyContext(); + throw; + } + } + + public ValueTask DisposeAsync() + { + return _core.DisposeAsync(); + } + + // A subscription is one logical attachment listening on the type's two delivery channels: the send (queue-role) + // destination and this subscriber's identity on the publish (topic-role) route. The publish channel is provisioned + // before listening so a publish can reach it from the first message. + private (ListenerConfig Send, ListenerConfig Publish) BuildChannels(MessageSubscriptionOptions? options, Type fallbackType) + { + options ??= new MessageSubscriptionOptions(); + if (options.PerInstance && !String.IsNullOrEmpty(options.Subscription)) + throw new ArgumentException("PerInstance and Subscription are mutually exclusive: PerInstance derives a unique per-instance subscription.", nameof(options)); + + var routeType = options.RouteType ?? fallbackType; + + // The default consumer key is unique per subscription so multiple handlers can attach to the same type: they + // compete round-robin for sent messages (a command still reaches exactly one handler instance) and each keeps + // its own subscriber group for published ones. An explicit Key opts subscriptions into one shared group. + string uniqueKey = Guid.NewGuid().ToString("N"); + + var destination = GetDestination(routeType, options.Destination); + var send = new ListenerConfig + { + Source = destination, + Key = !String.IsNullOrEmpty(options.Key) ? options.Key : $"{destination.Key}:{uniqueKey}", + MessageType = routeType, + AckMode = options.AckMode, + MaxConcurrency = options.MaxConcurrency, + MaxAttempts = options.MaxAttempts, + RedeliveryBackoff = options.RedeliveryBackoff, + DeadLetterWhen = options.DeadLetterWhen + }; + + var topic = GetTopic(routeType, options.Topic); + string subscription = options.PerInstance + ? $"{Environment.MachineName}-{Guid.NewGuid():N}" + : options.Subscription ?? QualifySubscription(GetSubscription(routeType, topic.Name, null), options.SubscriptionQualifier); + var publish = new ListenerConfig + { + // The source is the topic-qualified subscription address, not the bare subscription name, so the same + // subscription identity used on two topics resolves to two distinct sources (and isolates). + Source = DestinationAddress.ForSubscription(topic.Name, subscription), + Key = !String.IsNullOrEmpty(options.Key) ? options.Key : $"{topic.Name}:{subscription}:{uniqueKey}", + MessageType = routeType, + AckMode = options.AckMode, + MaxConcurrency = options.MaxConcurrency, + MaxAttempts = options.MaxAttempts, + RedeliveryBackoff = options.RedeliveryBackoff, + DeadLetterWhen = options.DeadLetterWhen + }; + + return (send, publish); + } + + private static string QualifySubscription(string identity, string? qualifier) + { + return String.IsNullOrEmpty(qualifier) ? identity : $"{identity}.{MessageRoutingConventions.ToKebabCase(qualifier)}"; + } + + // Delivery semantics must never be invisible: log each subscription's effective topology (which destination it + // consumes, which subscriber group it joins, and its retry posture) once at subscribe time. + private void LogSubscription(ListenerConfig send, ListenerConfig publish, bool sentWired, bool publishedWired) + { + _logger.LogInformation( + "Subscribed {MessageType}: send={Destination}, publish={Subscription}, concurrency={MaxConcurrency}, attempts={MaxAttempts}, ack={AckMode}", + send.MessageType.Name, sentWired ? send.Source.Key : "(none)", publishedWired ? publish.Source.Key : "(none)", Math.Max(1, send.MaxConcurrency), send.MaxAttempts?.ToString() ?? "default", send.AckMode); + } + + private Task EnsureTopicAsync(DestinationAddress topic, CancellationToken cancellationToken) + { + return _core.EnsureAsync([new DestinationDeclaration { Address = topic }], cancellationToken); + } + + private Task EnsureSubscriptionAsync(ListenerConfig config, CancellationToken cancellationToken) + { + return _core.EnsureAsync([ + new DestinationDeclaration { Address = DestinationAddress.ForTopic(config.Source.Topic!) }, + new DestinationDeclaration { Address = config.Source } + ], cancellationToken); + } + + private DestinationAddress GetDestination(Type messageType, string? destination) + { + return DestinationAddress.ForQueue(_core.Router.ResolveRoute(new MessageRouteContext + { + MessageType = messageType, + Role = MessageRouteRole.QueueDestination, + OperationOverride = destination + })); + } + + private DestinationAddress GetTopic(Type messageType, string? topic) + { + return DestinationAddress.ForTopic(_core.Router.ResolveRoute(new MessageRouteContext + { + MessageType = messageType, + Role = MessageRouteRole.PubSubTopic, + OperationOverride = topic + })); + } + + private string GetSubscription(Type messageType, string topic, string? subscription) + { + return _core.Router.ResolveSubscription(new MessageSubscriptionContext + { + MessageType = messageType, + Topic = topic, + OperationOverride = subscription + }); + } + + private static MessageEnvelopeOptions ToEnvelope(MessageSendOptions options) + { + return new MessageEnvelopeOptions + { + Priority = options.Priority, + Delay = options.Delay, + DeliverAt = options.DeliverAt, + TimeToLive = options.TimeToLive, + CorrelationId = options.CorrelationId, + Headers = options.Headers + }; + } + + private static MessageEnvelopeOptions ToEnvelope(MessagePublishOptions options) + { + return new MessageEnvelopeOptions + { + Priority = options.Priority, + Delay = options.Delay, + DeliverAt = options.DeliverAt, + TimeToLive = options.TimeToLive, + CorrelationId = options.CorrelationId, + Headers = options.Headers + }; + } + + private sealed class MessageSubscription : IMessageSubscription + { + private readonly MessageListenerHandle? _sent; + private readonly MessageListenerHandle? _published; + + public MessageSubscription(MessageListenerHandle? sent, MessageListenerHandle? published) + { + _sent = sent; + _published = published; + } + + public string Key => (_sent ?? _published)!.Key; + public string Destination => _sent?.Source.Key ?? ""; + public string Topic => _published?.Topic ?? ""; + public string Subscription => _published?.Subscription ?? ""; + public string Source => _published?.Source.Key ?? ""; + + public async ValueTask DisposeAsync() + { + if (_sent is not null) + await _sent.DisposeAsync().AnyContext(); + if (_published is not null) + await _published.DisposeAsync().AnyContext(); + } + } +} diff --git a/src/Foundatio/Messaging/MessageBusBase.cs b/src/Foundatio/Messaging/MessageBusBase.cs index aff97ce4b..85b1943b2 100644 --- a/src/Foundatio/Messaging/MessageBusBase.cs +++ b/src/Foundatio/Messaging/MessageBusBase.cs @@ -13,7 +13,7 @@ using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; -namespace Foundatio.Messaging; +namespace Foundatio.Messaging.Legacy; public abstract class MessageBusBase : IMessageBus, IHaveLogger, IHaveLoggerFactory, IHaveTimeProvider, IHaveResiliencePolicyProvider, IDisposable, IAsyncDisposable where TOptions : SharedMessageBusOptions { diff --git a/src/Foundatio/Messaging/MessageClientCore.cs b/src/Foundatio/Messaging/MessageClientCore.cs new file mode 100644 index 000000000..8e8c8f0d6 --- /dev/null +++ b/src/Foundatio/Messaging/MessageClientCore.cs @@ -0,0 +1,1295 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Diagnostics; +using System.Diagnostics.Metrics; +using System.Globalization; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Serializer; +using Foundatio.Utility; +using Microsoft.Extensions.Logging; + +namespace Foundatio.Messaging; + +/// +/// Core-owned messaging instruments. Counters and histograms are transport-agnostic and shared by every +/// instance so that send/receive/settlement volume and handler +/// latency are observable regardless of which transport is plugged in. +/// +internal static class MessagingInstruments +{ + public static readonly Counter Sent = FoundatioDiagnostics.Meter.CreateCounter("foundatio.messaging.sent", description: "Number of messages sent to a destination"); + public static readonly Counter Received = FoundatioDiagnostics.Meter.CreateCounter("foundatio.messaging.received", description: "Number of messages received from a source"); + public static readonly Counter Completed = FoundatioDiagnostics.Meter.CreateCounter("foundatio.messaging.completed", description: "Number of messages completed"); + public static readonly Counter Abandoned = FoundatioDiagnostics.Meter.CreateCounter("foundatio.messaging.abandoned", description: "Number of messages abandoned"); + public static readonly Counter DeadLettered = FoundatioDiagnostics.Meter.CreateCounter("foundatio.messaging.deadlettered", description: "Number of messages dead-lettered"); + public static readonly Counter Unhandled = FoundatioDiagnostics.Meter.CreateCounter("foundatio.messaging.unhandled", description: "Number of received messages with no registered consumer for their type"); + public static readonly Histogram HandlerTime = FoundatioDiagnostics.Meter.CreateHistogram("foundatio.messaging.handlertime", unit: "ms", description: "Message handler execution time"); +} + +/// +/// Transport-neutral envelope options shared by queue send and pub/sub publish operations. +/// +internal sealed record MessageEnvelopeOptions +{ + public MessagePriority Priority { get; init; } = MessagePriority.Normal; + public TimeSpan? Delay { get; init; } + public DateTimeOffset? DeliverAt { get; init; } + public TimeSpan? TimeToLive { get; init; } + public string? CorrelationId { get; init; } + public MessageHeaders? Headers { get; init; } +} + +/// +/// Describes a consumer/subscription listener independent of whether it is backed by a queue or a pub/sub subscription. +/// +internal sealed record ListenerConfig +{ + public required DestinationAddress Source { get; init; } + public required string Key { get; init; } + public required Type MessageType { get; init; } + public AckMode AckMode { get; init; } = AckMode.Auto; + public int MaxConcurrency { get; init; } = 1; + // Null falls back to the client's default RetryPolicy. + public int? MaxAttempts { get; init; } + public Func? RedeliveryBackoff { get; init; } + public Func? DeadLetterWhen { get; init; } +} + +/// +/// Shared implementation behind : serialization, header/trace +/// construction, routing-agnostic send (with batch chunking and runtime-store scheduled dispatch), received-message +/// creation with poison handling, auto/manual ack settlement, and the resilient consumer/subscription loop. +/// +internal sealed class MessageClientCore : IAsyncDisposable +{ + private readonly IMessageTransport _transport; + private readonly ISerializer _serializer; + private readonly IMessageRouter _router; + private readonly IScheduledDispatchStore? _runtimeStore; + private readonly TimeProvider _timeProvider; + private readonly ILogger _logger; + private readonly Func _exceptionFactory; + private readonly RetryPolicy _retryPolicy; + private readonly IMessageTypeRegistry _typeRegistry; + private readonly string? _contentType; + private readonly bool _ownsTransport; + private readonly ConcurrentDictionary _sources = new(); + private readonly TopologyMode _topologyMode; + private readonly ConcurrentDictionary _validatedDestinations = new(); + private int _isDisposed; + + public MessageClientCore(IMessageTransport transport, ISerializer serializer, IMessageRouter router, + IScheduledDispatchStore? runtimeStore, TimeProvider timeProvider, ILogger logger, Func exceptionFactory, RetryPolicy? retryPolicy = null, bool ownsTransport = true, IMessageTypeRegistry? typeRegistry = null, string? contentType = null, TopologyMode topologyMode = TopologyMode.Ensure) + { + _topologyMode = topologyMode; + _transport = transport ?? throw new ArgumentNullException(nameof(transport)); + _serializer = serializer; + _router = router; + _runtimeStore = runtimeStore; + _timeProvider = timeProvider; + _logger = logger; + _exceptionFactory = exceptionFactory; + _retryPolicy = retryPolicy ?? new RetryPolicy(); + _typeRegistry = typeRegistry ?? new MessageTypeRegistry(); + _contentType = contentType; + _ownsTransport = ownsTransport; + } + + public IMessageRouter Router => _router; + + // A transport that does not advertise ITransportInfo is assumed to support every role (test doubles, minimal + // providers); one that does advertise is held to its declaration. + public bool SupportsRole(DestinationRole role) + { + return _transport is not ITransportInfo info || info.SupportedRoles.Contains(role); + } + + public Task EnsureAsync(IReadOnlyList declarations, CancellationToken cancellationToken) + { + return _topologyMode switch + { + TopologyMode.None => Task.CompletedTask, + TopologyMode.Validate => ValidateDeclarationsAsync(declarations, cancellationToken), + _ => _transport is ISupportsProvisioning provisioning + ? provisioning.EnsureAsync(declarations, cancellationToken) + : Task.CompletedTask + }; + } + + // Validate never creates: each destination is checked once (successes are cached so steady-state publishes pay no + // exists round-trip) and a missing one fails loudly instead of being silently created on a broker the app is not + // supposed to administer. + private async Task ValidateDeclarationsAsync(IReadOnlyList declarations, CancellationToken cancellationToken) + { + if (_transport is not ISupportsProvisioning provisioning) + throw new NotSupportedException($"{nameof(TopologyMode)}.{nameof(TopologyMode.Validate)} requires a transport that can check destination existence; \"{_transport.GetType().Name}\" does not support provisioning. Use {nameof(TopologyMode)}.{nameof(TopologyMode.None)} when the transport cannot inspect a pre-provisioned broker."); + + foreach (var declaration in declarations) + { + if (_validatedDestinations.ContainsKey(declaration.Address)) + continue; + + if (!await provisioning.ExistsAsync(declaration.Address, cancellationToken).AnyContext()) + throw _exceptionFactory($"Message topology destination {declaration.Address} does not exist and {nameof(TopologyMode)}.{nameof(TopologyMode.Validate)} never creates topology. Provision it out of band or use {nameof(TopologyMode)}.{nameof(TopologyMode.Ensure)}.", null); + + _validatedDestinations.TryAdd(declaration.Address, 0); + } + } + + public async Task SendAsync(ScheduledDispatchKind kind, Type messageType, object message, MessageEnvelopeOptions options, DestinationAddress destination, Func? ensureDestination, CancellationToken cancellationToken) + { + ThrowIfDisposed(); + ValidateCapabilities(destination.Role, options.Priority, options.TimeToLive); + + var sendOptions = BuildSendOptions(options); + string messageId = Guid.NewGuid().ToString("N"); + var transportMessage = CreateTransportMessage(message, messageType, options, messageId); + + if (ensureDestination is not null) + await ensureDestination(destination, cancellationToken).AnyContext(); + + if (await TryScheduleAsync(kind, destination, [transportMessage], sendOptions, cancellationToken).AnyContext()) + return messageId; + + // Send is throw-on-failure: SendChunkedAsync propagates any transport error, so a returned result means the + // message was accepted. Fall back to the pre-assigned id if the transport reported none. + var items = await SendChunkedAsync(destination, [transportMessage], sendOptions, cancellationToken).AnyContext(); + return (items.Count > 0 ? items[0].MessageId : null) ?? messageId; + } + + public async Task SendBatchAsync(ScheduledDispatchKind kind, IEnumerable messages, Type? declaredType, MessageEnvelopeOptions options, Func resolveDestination, Func? ensureDestination, CancellationToken cancellationToken) + { + ThrowIfDisposed(); + ValidateCapabilities(RoleFor(kind), options.Priority, options.TimeToLive); + + var sendOptions = BuildSendOptions(options); + var grouped = new Dictionary>(); + + foreach (var message in messages) + { + ArgumentNullException.ThrowIfNull(message); + Type messageType = declaredType ?? message.GetType(); + var destination = resolveDestination(messageType); + + if (!grouped.TryGetValue(destination, out var transportMessages)) + { + transportMessages = []; + grouped.Add(destination, transportMessages); + } + + transportMessages.Add(CreateTransportMessage(message, messageType, options, messageId: null)); + } + + foreach (var group in grouped) + { + if (ensureDestination is not null) + await ensureDestination(group.Key, cancellationToken).AnyContext(); + + if (await TryScheduleAsync(kind, group.Key, group.Value, sendOptions, cancellationToken).AnyContext()) + continue; + + // Send is throw-on-failure (SendChunkedAsync propagates any transport error); a returned result means all + // messages in this destination group were accepted. + await SendChunkedAsync(group.Key, group.Value, sendOptions, cancellationToken).AnyContext(); + } + } + + public Task StartListenerAsync(ListenerConfig config, Func handler, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(handler); + return RegisterConsumerAsync(config, handler, async (entry, token) => + { + var received = CreateMessageContext(entry, token); + await HandleMessageAsync(received, config, handler, token).AnyContext(); + }, cancellationToken); + } + + public Task StartListenerAsync(ListenerConfig config, Func, CancellationToken, Task> handler, CancellationToken cancellationToken) where T : class + { + ArgumentNullException.ThrowIfNull(handler); + return RegisterConsumerAsync(config, handler, async (entry, token) => + { + var received = await CreateMessageContextAsync(entry, token).AnyContext(); + await HandleMessageAsync(received, config, handler, token).AnyContext(); + }, cancellationToken); + } + + public async ValueTask DisposeAsync() + { + if (Interlocked.Exchange(ref _isDisposed, 1) == 1) + return; + + foreach (var listener in _sources.Values.ToArray()) + await listener.DisposeAsync().AnyContext(); + + // Only dispose the transport when this client owns it. In DI the transport is a shared singleton owned by the + // container, so neither the queue nor the pub/sub client should dispose it (that would double-dispose the one + // the other still depends on). + if (_ownsTransport) + await _transport.DisposeAsync().AnyContext(); + } + + // Multiple typed consumers can share one destination. They attach to a single per-source listener whose loop + // demultiplexes each message to the consumer registered for its type; a type with no registered consumer is + // handled by HandleUnmatchedAsync. Starting the same consumer key with a matching handler/options is idempotent. + private async Task RegisterConsumerAsync(ListenerConfig config, Delegate handler, Func dispatch, CancellationToken cancellationToken) + { + ThrowIfDisposed(); + cancellationToken.ThrowIfCancellationRequested(); + + bool catchAll = IsCatchAll(config.MessageType); + var registration = new ConsumerRegistration + { + Key = config.Key, + Config = config, + Dispatch = dispatch, + Info = MessageListenerRegistration.Create(handler, config), + IsCatchAll = catchAll, + TypeName = catchAll ? null : _typeRegistry.GetName(config.MessageType) + }; + + while (true) + { + var listener = _sources.GetOrAdd(config.Source, source => new SourceListener(this, source)); + if (listener.TryAddConsumer(registration, out var handle, out bool created)) + { + if (created) + { + try + { + await listener.StartAsync(cancellationToken).AnyContext(); + } + catch + { + await listener.DisposeAsync().AnyContext(); + throw; + } + } + + return handle; + } + + // The listener was disposing as its last consumer detached; drop our stale reference and retry. + _sources.TryRemove(new KeyValuePair(config.Source, listener)); + } + } + + // A concrete message type binds an exact-type consumer; object/interface/abstract route types are catch-alls that + // receive every message a more specific typed consumer did not claim (the grouped/raw-envelope path). + private static bool IsCatchAll(Type messageType) + { + return messageType == typeof(object) || messageType.IsInterface || messageType.IsAbstract; + } + + private async Task HandleUnmatchedAsync(TransportEntry entry, DestinationAddress source, CancellationToken cancellationToken) + { + MessagingInstruments.Unhandled.Add(1, new KeyValuePair("source", source)); + + var message = CreateMessageContext(entry, cancellationToken); + + // Same WARN-while-retryable / ERROR-when-terminal convention as handler failures. + if (message.Attempts >= _retryPolicy.UnmatchedMaxAttempts) + _logger.LogError("No consumer registered for message type \"{MessageType}\" on \"{Source}\" (attempt {Attempt} of {MaxAttempts}); dead-lettering as no-handler", message.MessageType, source, message.Attempts, _retryPolicy.UnmatchedMaxAttempts); + else + _logger.LogWarning("No consumer registered for message type \"{MessageType}\" on \"{Source}\" (attempt {Attempt} of {MaxAttempts}); will retry", message.MessageType, source, message.Attempts, _retryPolicy.UnmatchedMaxAttempts); + + // Retry so a node that does handle this type can pick it up; dead-letter as "no-handler" once the lenient + // budget is exhausted so a genuinely orphaned type cannot loop forever. + await SettleFailedMessageAsync(message, unrecoverable: false, _retryPolicy.UnmatchedMaxAttempts, _retryPolicy.UnmatchedBackoff, deadLetterReason: "no-handler", exception: null, cancellationToken).AnyContext(); + + // Surface to direct callers. The throw is caught (and not re-logged) by the loop's per-message handling + // (SafeProcessAsync), so it never tears down the receive loop or the other type handlers sharing this source. + throw new UnhandledMessageTypeException(message.MessageType, source.Key); + } + + // MaxConcurrency bounds the number of in-flight messages. A slot is held from receive until the message settles + // and is released the instant that one message finishes — so a single slow message never stalls the other slots + // (no head-of-line blocking) and steady-state utilization stays at the configured concurrency. A failure while + // receiving or while processing a single entry (including a poison message that was already dead-lettered) must + // never tear down the loop, otherwise one bad message or a transient transport blip silently stops consumption. + private async Task RunPullLoopAsync(DestinationAddress source, ISupportsPull pull, Func onMessage, int maxConcurrency, CancellationToken cancellationToken) + { + maxConcurrency = Math.Max(1, maxConcurrency); + var slots = new SemaphoreSlim(maxConcurrency, maxConcurrency); + var inFlight = new ConcurrentDictionary(); + + try + { + while (!cancellationToken.IsCancellationRequested) + { + // Block for a free slot before receiving so we never pull more than we can process concurrently. + try + { + await slots.WaitAsync(cancellationToken).AnyContext(); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + break; + } + + // Opportunistically claim any other idle slots so a transport that supports batch receive can still + // pull a batch while keeping per-message slot release. WaitAsync(Zero) is a non-blocking try-acquire. + int claimed = 1; + while (claimed < maxConcurrency && await slots.WaitAsync(TimeSpan.Zero).AnyContext()) + claimed++; + + var pollWindow = TimeSpan.FromSeconds(1); + long pollStart = _timeProvider.GetTimestamp(); + IReadOnlyList entries; + try + { + entries = await pull.ReceiveAsync(source, new ReceiveRequest + { + MaxMessages = claimed, + MaxWaitTime = pollWindow + }, cancellationToken).AnyContext(); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + ReleaseSlots(slots, claimed); + break; + } + catch (Exception ex) + { + ReleaseSlots(slots, claimed); + _logger.LogError(ex, "Error receiving from \"{Source}\"; retrying: {Message}", source, ex.Message); + await _timeProvider.SafeDelay(TimeSpan.FromSeconds(1), cancellationToken).AnyContext(); + continue; + } + + // We hold exactly `claimed` slots and release one per processed entry, so never process more than we + // claimed: a well-behaved transport returns <= MaxMessages, but a transport that ignores MaxMessages and + // over-returns would otherwise release more slots than acquired (breaching the cap / overflowing the + // semaphore). Any over-returned entries are left unsettled and redeliver after their visibility window. + int toProcess = Math.Min(entries.Count, claimed); + ReleaseSlots(slots, claimed - toProcess); // return slots we claimed but won't fill (always >= 0) + + // An empty poll should have blocked for MaxWaitTime; a transport that returns empty early (or + // synchronously) would otherwise hot-spin this loop, so sleep out the remainder of the window. + if (toProcess == 0) + { + var remaining = pollWindow - _timeProvider.GetElapsedTime(pollStart); + if (remaining > TimeSpan.Zero) + await _timeProvider.SafeDelay(remaining, cancellationToken).AnyContext(); + } + + for (int index = 0; index < toProcess; index++) + { + var task = ProcessAndReleaseSlotAsync(entries[index], onMessage, source, slots, cancellationToken); + if (!task.IsCompleted) + { + inFlight[task] = 0; + _ = task.ContinueWith(static (t, state) => ((ConcurrentDictionary)state!).TryRemove(t, out _), inFlight, TaskScheduler.Default); + } + } + } + } + finally + { + // Drain in-flight handlers before the semaphore is disposed so their slot releases never hit a disposed handle. + await Task.WhenAll(inFlight.Keys.ToArray()).AnyContext(); + slots.Dispose(); + } + } + + private async Task ProcessAndReleaseSlotAsync(TransportEntry entry, Func onMessage, DestinationAddress source, SemaphoreSlim slots, CancellationToken cancellationToken) + { + try + { + await SafeProcessAsync(entry, onMessage, source, cancellationToken).AnyContext(); + } + finally + { + slots.Release(); + } + } + + private static void ReleaseSlots(SemaphoreSlim slots, int count) + { + if (count > 0) + slots.Release(count); + } + + private async Task SafeProcessAsync(TransportEntry entry, Func onMessage, DestinationAddress source, CancellationToken cancellationToken) + { + try + { + await onMessage(entry, cancellationToken).AnyContext(); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + } + catch (UnhandledMessageTypeException) + { + // Already settled AND classified (WARN-retryable / ERROR-terminal) by HandleUnmatchedAsync; re-logging + // here would emit an ERROR for every retryable attempt. + } + catch (Exception ex) + { + // The message has already been settled (dead-lettered on deserialize failure, abandoned/dead-lettered on + // handler error); swallowing here keeps the loop alive for the next message. + _logger.LogError(ex, "Error processing message \"{MessageId}\" from \"{Source}\": {Message}", entry.Id, source, ex.Message); + } + } + + private async Task HandleMessageAsync(TMessage message, ListenerConfig config, Func handler, CancellationToken cancellationToken) where TMessage : IMessageContext + { + // Re-establish the producer's trace context on the consumer side so a cross-process trace continues here + // instead of breaking at the transport boundary. + using var activity = StartProcessActivity(message, config); + long startTimestamp = Stopwatch.GetTimestamp(); + try + { + await handler(message, cancellationToken).AnyContext(); + + if (config.AckMode == AckMode.Auto && !message.IsHandled) + await message.CompleteAsync(cancellationToken).AnyContext(); + } + catch (Exception ex) + { + activity?.SetErrorStatus(ex); + + // The handler already settled (e.g. terminal-rejected a poison payload, then rethrew): the settle path + // will skip, so don't log a retry/dead-letter that won't happen. + if (message.IsHandled) + { + _logger.LogWarning(ex, "Handler threw after settling message \"{MessageId}\" from \"{Source}\"; no further settlement will occur: {Message}", message.Id, config.Source, ex.Message); + return; + } + + int maxAttempts = config.MaxAttempts ?? _retryPolicy.MaxAttempts; + var backoff = config.RedeliveryBackoff ?? _retryPolicy.Backoff; + + bool unrecoverable = false; + try + { + unrecoverable = (config.DeadLetterWhen ?? _retryPolicy.DeadLetterWhen)?.Invoke(ex) == true; + } + catch (Exception predicateEx) + { + _logger.LogError(predicateEx, "DeadLetterWhen predicate threw for message \"{MessageId}\"; treating the failure as retryable: {Message}", message.Id, predicateEx.Message); + } + + // A retry that can still happen is a warning; the terminal decision (unrecoverable or attempts exhausted) + // is the error worth alerting on. + if (unrecoverable || message.Attempts >= maxAttempts) + _logger.LogError(ex, "Handler failed for message \"{MessageId}\" from \"{Source}\" (attempt {Attempt} of {MaxAttempts}); dead-lettering: {Message}", message.Id, config.Source, message.Attempts, maxAttempts, ex.Message); + else + _logger.LogWarning(ex, "Handler failed for message \"{MessageId}\" from \"{Source}\" (attempt {Attempt} of {MaxAttempts}); will retry: {Message}", message.Id, config.Source, message.Attempts, maxAttempts, ex.Message); + + string reason = unrecoverable ? $"unrecoverable:{ex.GetType().Name}" : "handler-error"; + await SettleFailedMessageAsync(message, unrecoverable, maxAttempts, backoff, reason, ex, cancellationToken).AnyContext(); + } + finally + { + MessagingInstruments.HandlerTime.Record(Stopwatch.GetElapsedTime(startTimestamp).TotalMilliseconds, new KeyValuePair("source", config.Source.Key)); + } + } + + private static Activity? StartProcessActivity(IMessageContext message, ListenerConfig config) + { + string? traceParent = message.Headers.GetValueOrDefault(KnownHeaders.TraceParent); + var activity = FoundatioDiagnostics.ActivitySource.StartActivity("ProcessMessage", ActivityKind.Consumer, traceParent); + if (activity is null) + return null; + + string? traceState = message.Headers.GetValueOrDefault(KnownHeaders.TraceState); + if (!String.IsNullOrEmpty(traceState)) + activity.TraceStateString = traceState; + + activity.DisplayName = $"Process: {message.MessageType ?? config.MessageType.Name}"; + + if (activity.IsAllDataRequested) + { + activity.SetTag("messaging.source", config.Source.Key); + activity.SetTag("messaging.message.id", message.Id); + } + + return activity; + } + + private static Task SettleFailedMessageAsync(IMessageContext message, bool unrecoverable, int maxAttempts, Func? backoff, string deadLetterReason, Exception? exception, CancellationToken cancellationToken) + { + if (message.IsHandled) + return Task.CompletedTask; + + if (unrecoverable || message.Attempts >= maxAttempts) + return message.RejectAsync(new RejectOptions { Terminal = true, Reason = deadLetterReason, Exception = exception }, cancellationToken); + + // Policy-driven delays are best-effort: a transport that can't honor the delay redelivers immediately rather + // than failing the settle (an explicit caller-requested delay stays strict). + return message.RejectAsync(new RejectOptions { RedeliveryDelay = backoff?.Invoke(message.Attempts), BestEffortDelay = true }, cancellationToken); + } + + private MessageContext CreateMessageContext(TransportEntry entry, CancellationToken cancellationToken) + { + MessagingInstruments.Received.Add(1, new KeyValuePair("source", entry.Destination.Key)); + return new MessageContext(_transport, entry, cancellationToken, _runtimeStore, _timeProvider, _retryPolicy.DeadLetterDestination, _logger); + } + + private async Task> CreateMessageContextAsync(TransportEntry entry, CancellationToken cancellationToken) where T : class + { + MessagingInstruments.Received.Add(1, new KeyValuePair("source", entry.Destination.Key)); + + // For an interface/base route the body cannot be deserialized as T directly. Resolve the concrete payload type + // from the message-type header via the registry and deserialize that, then hand it back as T (the concrete + // instance is assignable to T). Exact concrete routes deserialize as T directly. + Type targetType = typeof(T); + if (typeof(T).IsInterface || typeof(T).IsAbstract) + { + string? typeName = entry.Headers.GetValueOrDefault(KnownHeaders.MessageType); + var resolved = String.IsNullOrEmpty(typeName) ? null : _typeRegistry.Resolve(typeName); + if (resolved is null || !typeof(T).IsAssignableFrom(resolved)) + { + await DeadLetterPoisonMessageAsync(entry, "unresolved-type", exception: null, cancellationToken).AnyContext(); + throw _exceptionFactory($"Unable to resolve a concrete type \"{typeName}\" assignable to \"{typeof(T).Name}\" for message \"{entry.Id}\".", null); + } + + targetType = resolved; + } + + T? message; + try + { + message = _serializer.Deserialize(entry.Body, targetType) as T; + } + catch (Exception ex) + { + await DeadLetterPoisonMessageAsync(entry, "deserialize-failure", ex, cancellationToken).AnyContext(); + throw _exceptionFactory($"Unable to deserialize message \"{entry.Id}\".", ex); + } + + if (message is null) + { + await DeadLetterPoisonMessageAsync(entry, "deserialize-failure", exception: null, cancellationToken).AnyContext(); + throw _exceptionFactory($"Message \"{entry.Id}\" deserialized to null.", null); + } + + return new MessageContext(_transport, entry, message, cancellationToken, _runtimeStore, _timeProvider, _retryPolicy.DeadLetterDestination, _logger); + } + + private Task DeadLetterPoisonMessageAsync(TransportEntry entry, string reason, Exception? exception, CancellationToken cancellationToken) + { + MessagingInstruments.DeadLettered.Add(1, new KeyValuePair("source", entry.Destination.Key)); + var enriched = entry with { Headers = MessageContext.BuildDeadLetterHeaders(entry, entry.DeliveryCount, exception, _timeProvider) }; + return MessageContext.DeadLetterOrDropAsync(_transport, enriched, reason, _retryPolicy.DeadLetterDestination, _logger, cancellationToken); + } + + + private TransportMessage CreateTransportMessage(object message, Type messageType, MessageEnvelopeOptions options, string? messageId) + { + // Content type is intentionally not written as a header: the receive path always uses the single configured + // serializer, so advertising a per-message content type would be misleading until real negotiation exists. + var headers = (options.Headers ?? MessageHeaders.Empty).ToBuilder() + .Set(KnownHeaders.MessageType, _typeRegistry.GetName(messageType)) + .Set(KnownHeaders.Priority, options.Priority.ToString()); + + if (!String.IsNullOrEmpty(options.CorrelationId)) + headers.Set(KnownHeaders.CorrelationId, options.CorrelationId); + + if (Activity.Current is { } activity) + { + if (!String.IsNullOrEmpty(activity.Id)) + headers.SetIfMissing(KnownHeaders.TraceParent, activity.Id); + + if (!String.IsNullOrEmpty(activity.TraceStateString)) + headers.SetIfMissing(KnownHeaders.TraceState, activity.TraceStateString); + } + + if (options.TimeToLive is { } ttl) + headers.Set(KnownHeaders.Expiration, _timeProvider.GetUtcNow().Add(ttl).ToString("O", CultureInfo.InvariantCulture)); + + return new TransportMessage + { + Body = _serializer.SerializeToBytes(message), + Headers = headers.Build(), + MessageId = messageId, + ContentType = _contentType + }; + } + + // A pub/sub publish targets a topic; everything else targets a queue. Stating the role lets the transport route + // without inferring (e.g. SNS publish vs. SQS send). + private static DestinationRole RoleFor(ScheduledDispatchKind kind) + { + return kind == ScheduledDispatchKind.PubSubMessage ? DestinationRole.Topic : DestinationRole.Queue; + } + + private TransportSendOptions BuildSendOptions(MessageEnvelopeOptions options) + { + return new TransportSendOptions + { + Priority = options.Priority, + DeliverAt = options.DeliverAt ?? (options.Delay is { } delay ? _timeProvider.GetUtcNow().Add(delay) : null) + }; + } + + // Capabilities are role-aware: the same transport can honor a feature on queues but not topics (SQS DelaySeconds + // vs. SNS publish), so every send-path decision asks for the destination role it is actually targeting. + private TransportCapabilities CapabilitiesFor(DestinationRole role) + { + return _transport is ITransportInfo info ? info.GetCapabilities(role) : TransportCapabilities.None; + } + + private void ValidateCapabilities(DestinationRole role, MessagePriority priority, TimeSpan? timeToLive) + { + var capabilities = CapabilitiesFor(role); + + if (priority != MessagePriority.Normal && !capabilities.Priority) + throw new NotSupportedException($"Transport \"{_transport.GetType().Name}\" does not support message priority for {role} destinations."); + + if (timeToLive is not null && !capabilities.Expiration) + throw new NotSupportedException($"Transport \"{_transport.GetType().Name}\" does not support message expiration for {role} destinations."); + } + + private async Task TryScheduleAsync(ScheduledDispatchKind kind, DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken cancellationToken) + { + if (!ShouldScheduleThroughRuntimeStore(destination.Role, options, out var dueUtc)) + return false; + + foreach (var message in messages) + { + string messageId = message.MessageId ?? Guid.NewGuid().ToString("N"); + await _runtimeStore!.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = messageId, + Kind = kind, + Destination = destination, + Body = message.Body, + Headers = message.Headers, + Options = options with { DeliverAt = null }, + DueUtc = dueUtc + }, cancellationToken).AnyContext(); + } + + return true; + } + + private bool ShouldScheduleThroughRuntimeStore(DestinationRole role, TransportSendOptions options, out DateTimeOffset dueUtc) + { + dueUtc = options.DeliverAt.GetValueOrDefault(); + var now = _timeProvider.GetUtcNow(); + if (options.DeliverAt is null || dueUtc <= now) + return false; + + // A destination can deliver natively only up to its advertised maximum; a delay longer than the broker supports + // (e.g. SQS caps DelaySeconds at 15 minutes) must route through the durable runtime store rather than be + // silently truncated to the broker's ceiling. The check is per destination role: a transport whose queues take + // a native delay may still have topics that cannot (SQS vs. SNS), and those publishes must fall back too. + var capabilities = CapabilitiesFor(role); + if (capabilities.DelayedDelivery && (capabilities.MaxDeliveryDelay is not { } max || dueUtc - now <= max)) + return false; + + if (_runtimeStore is null) + throw _exceptionFactory($"Delayed delivery requires either native delayed-delivery support from transport \"{_transport.GetType().Name}\" for {role} destinations (within its supported maximum) or a registered job runtime store.", null); + + return true; + } + + private async Task> SendChunkedAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken cancellationToken) + { + var capabilities = CapabilitiesFor(destination.Role); + + // Enforce a transport-declared maximum message size up front with a clear error, rather than letting an opaque + // broker rejection surface mid-send (the limit is advertised, so honor it). + if (capabilities.MaxMessageBytes is { } maxBytes) + { + foreach (var message in messages) + { + if (message.Body.Length > maxBytes) + throw _exceptionFactory($"Message of {message.Body.Length} bytes exceeds transport \"{_transport.GetType().Name}\" maximum of {maxBytes} bytes for destination \"{destination}\".", null); + } + } + + // Respect a transport-declared maximum batch size by splitting oversized sends into chunks. + int? maxBatchSize = capabilities.MaxBatchSize; + if (maxBatchSize is not { } limit || limit <= 0 || messages.Count <= limit) + { + var result = await _transport.SendAsync(destination, messages, options, cancellationToken).AnyContext(); + RecordSent(destination, result.Items); + return result.Items; + } + + var items = new List(messages.Count); + for (int offset = 0; offset < messages.Count; offset += limit) + { + var chunk = messages.Skip(offset).Take(limit).ToArray(); + var result = await _transport.SendAsync(destination, chunk, options, cancellationToken).AnyContext(); + RecordSent(destination, result.Items); + items.AddRange(result.Items); + } + + return items; + } + + private static void RecordSent(DestinationAddress destination, IReadOnlyList items) + { + // Every returned item was accepted (send is throw-on-failure). + if (items.Count > 0) + MessagingInstruments.Sent.Add(items.Count, new KeyValuePair("destination", destination.Key)); + } + + private ISupportsPull RequirePull() + { + return _transport as ISupportsPull + ?? throw _exceptionFactory($"Transport \"{_transport.GetType().Name}\" does not support pull receive.", null); + } + + private void RemoveSource(DestinationAddress source, SourceListener listener) + { + _sources.TryRemove(new KeyValuePair(source, listener)); + } + + private void ThrowIfDisposed() + { + ObjectDisposedException.ThrowIf(Volatile.Read(ref _isDisposed) == 1, this); + } + + private sealed class ConsumerRegistration + { + public required string Key { get; init; } + public required ListenerConfig Config { get; init; } + public required Func Dispatch { get; init; } + public required MessageListenerRegistration Info { get; init; } + public required bool IsCatchAll { get; init; } + public required string? TypeName { get; init; } + } + + // One receive loop per source. Consumers register by message type; the loop reads the message-type header and + // dispatches each entry to a consumer for that type (round-robin when several share a type, so same-type consumers + // compete), to the catch-all group for unmapped types, or to HandleUnmatchedAsync when nothing claims the type. + // The loop runs while at least one consumer is attached and shuts down when the last one detaches. + private sealed class SourceListener + { + private readonly MessageClientCore _core; + private readonly DestinationAddress _source; + private readonly object _lock = new(); + private readonly CancellationTokenSource _cancellationTokenSource = new(); + private readonly ConcurrentDictionary _consumers = new(StringComparer.Ordinal); + private readonly ConcurrentDictionary _byType = new(StringComparer.Ordinal); + private readonly ConsumerGroup _catchAll = new(); + private int _maxConcurrency = 1; + private IPushSubscription? _pushSubscription; + private Task? _loop; + private bool _isDisposed; + + public SourceListener(MessageClientCore core, DestinationAddress source) + { + _core = core; + _source = source; + } + + public bool TryAddConsumer(ConsumerRegistration registration, out MessageListenerHandle handle, out bool created) + { + handle = null!; + created = false; + + lock (_lock) + { + if (_isDisposed) + return false; + + if (_consumers.TryGetValue(registration.Key, out var existing)) + { + if (!existing.Registration.Info.Matches(registration.Info)) + throw new InvalidOperationException($"A consumer with key \"{registration.Key}\" is already registered with a different handler or options."); + + handle = existing.Handle; // idempotent re-registration + return true; + } + + int desired = Math.Max(1, registration.Config.MaxConcurrency); + if (_consumers.IsEmpty) + { + _maxConcurrency = desired; + created = true; + } + else if (desired != _maxConcurrency) + { + throw new InvalidOperationException($"Source \"{_source}\" is already consumed with MaxConcurrency {_maxConcurrency}; a conflicting MaxConcurrency {desired} was requested. Consumers sharing a destination must use the same MaxConcurrency."); + } + + handle = new MessageListenerHandle(_source, registration.Key, () => RemoveConsumerAsync(registration.Key)); + _consumers[registration.Key] = new Registered(registration, handle); + GroupFor(registration).Add(registration); + + return true; + } + } + + private ConsumerGroup GroupFor(ConsumerRegistration registration) + { + return registration.IsCatchAll ? _catchAll : _byType.GetOrAdd(registration.TypeName!, _ => new ConsumerGroup()); + } + + public async Task StartAsync(CancellationToken cancellationToken) + { + if (_core._transport is ISupportsPush push) + { + // Route the push callback through SafeProcessAsync so a throw (including an unmatched-type throw) is + // isolated to the message and never tears down the subscription. + _pushSubscription = await push.SubscribeAsync(_source, (entry, token) => _core.SafeProcessAsync(entry, DispatchAsync, _source, token), new PushOptions { MaxConcurrentMessages = Math.Max(1, _maxConcurrency) }, cancellationToken).AnyContext(); + return; + } + + if (_core._transport is not ISupportsPull pull) + throw _core._exceptionFactory($"Transport \"{_core._transport.GetType().Name}\" does not support receiving messages.", null); + + // Task.Run so a transport whose ReceiveAsync completes synchronously can never run the loop inline on the + // caller's thread and block the subscribe call from returning. + _loop = Task.Run(() => _core.RunPullLoopAsync(_source, pull, DispatchAsync, _maxConcurrency, _cancellationTokenSource.Token), CancellationToken.None); + } + + public async ValueTask DisposeAsync() + { + lock (_lock) + { + if (_isDisposed) + return; + + _isDisposed = true; + } + + await ShutdownAsync().AnyContext(); + } + + private async ValueTask RemoveConsumerAsync(string key) + { + bool shutdown = false; + lock (_lock) + { + if (!_consumers.TryRemove(key, out var registered)) + return; + + var registration = registered.Registration; + if (registration.IsCatchAll) + { + _catchAll.Remove(registration); + } + else if (registration.TypeName is { } typeName && _byType.TryGetValue(typeName, out var group)) + { + group.Remove(registration); + if (group.IsEmpty) + _byType.TryRemove(new KeyValuePair(typeName, group)); + } + + if (_consumers.IsEmpty && !_isDisposed) + { + _isDisposed = true; + shutdown = true; + } + } + + if (shutdown) + await ShutdownAsync().AnyContext(); + } + + private async Task ShutdownAsync() + { + await _cancellationTokenSource.CancelAsync().AnyContext(); + + if (_pushSubscription is not null) + await _pushSubscription.DisposeAsync().AnyContext(); + + if (_loop is not null) + { + try + { + await _loop.AnyContext(); + } + catch (OperationCanceledException) { } + } + + _cancellationTokenSource.Dispose(); + _core.RemoveSource(_source, this); + } + + private async Task DispatchAsync(TransportEntry entry, CancellationToken token) + { + var registration = Resolve(entry); + if (registration is null) + { + await _core.HandleUnmatchedAsync(entry, _source, token).AnyContext(); + return; + } + + await registration.Dispatch(entry, token).AnyContext(); + } + + private ConsumerRegistration? Resolve(TransportEntry entry) + { + string? typeName = entry.Headers.GetValueOrDefault(KnownHeaders.MessageType); + if (typeName is not null && _byType.TryGetValue(typeName, out var group) && group.Next() is { } typed) + return typed; + + return _catchAll.Next(); + } + + private sealed record Registered(ConsumerRegistration Registration, MessageListenerHandle Handle); + + // Consumers sharing a message type (or the catch-all) on one source compete: each message is dispatched to one + // of them, round-robin. The registration array is swapped under the listener lock; Next() reads it lock-free. + private sealed class ConsumerGroup + { + private ConsumerRegistration[] _registrations = []; + private int _next; + + public bool IsEmpty => Volatile.Read(ref _registrations).Length == 0; + + public void Add(ConsumerRegistration registration) + { + var current = _registrations; + var updated = new ConsumerRegistration[current.Length + 1]; + Array.Copy(current, updated, current.Length); + updated[^1] = registration; + Volatile.Write(ref _registrations, updated); + } + + public void Remove(ConsumerRegistration registration) + { + var current = _registrations; + int index = Array.IndexOf(current, registration); + if (index < 0) + return; + + var updated = new ConsumerRegistration[current.Length - 1]; + Array.Copy(current, 0, updated, 0, index); + Array.Copy(current, index + 1, updated, index, current.Length - index - 1); + Volatile.Write(ref _registrations, updated); + } + + public ConsumerRegistration? Next() + { + var snapshot = Volatile.Read(ref _registrations); + if (snapshot.Length == 0) + return null; + if (snapshot.Length == 1) + return snapshot[0]; + + int index = (int)((uint)Interlocked.Increment(ref _next) % (uint)snapshot.Length); + return snapshot[index]; + } + } + } +} + +internal class MessageContext : IMessageContext +{ + private readonly IMessageTransport _transport; + private readonly TransportEntry _entry; + private readonly IScheduledDispatchStore? _runtimeStore; + private readonly TimeProvider _timeProvider; + private readonly string? _deadLetterDestination; + private readonly ILogger _logger; + private int _isHandled; + + public MessageContext(IMessageTransport transport, TransportEntry entry, CancellationToken cancellationToken, IScheduledDispatchStore? runtimeStore = null, TimeProvider? timeProvider = null, string? deadLetterDestination = null, ILogger? logger = null) + { + _transport = transport; + _entry = entry; + _runtimeStore = runtimeStore; + _timeProvider = timeProvider ?? TimeProvider.System; + _deadLetterDestination = deadLetterDestination; + _logger = logger ?? Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance; + CancellationToken = cancellationToken; + } + + public string Id => _entry.Id; + public ReadOnlyMemory Body => _entry.Body; + public MessageHeaders Headers => _entry.Headers; + public string? CorrelationId => Headers.GetValueOrDefault(KnownHeaders.CorrelationId); + public string? MessageType => Headers.GetValueOrDefault(KnownHeaders.MessageType); + public MessagePriority Priority => Enum.TryParse(Headers.GetValueOrDefault(KnownHeaders.Priority), ignoreCase: true, out MessagePriority priority) ? priority : MessagePriority.Normal; + + // Reconcile the transport-reported delivery count with the message.attempts header. When redelivery-delay is + // served through the runtime-store fallback (transports without native ISupportsRedeliveryDelay), the message is + // re-sent as a brand-new transport message, so its DeliveryCount resets to 1; the carried-over attempt count + // lives in the header. Taking the max keeps MaxAttempts/dead-letter correct regardless of whether the transport + // honors the header, so the counter never silently resets and redelivery can't loop forever. + public int Attempts => Math.Max(_entry.DeliveryCount, ParseAttemptsHeader(_entry.Headers)); + public bool IsHandled => Volatile.Read(ref _isHandled) == 1; + public CancellationToken CancellationToken { get; } + + public Task CompleteAsync(CancellationToken cancellationToken = default) + { + if (!TryMarkHandled()) + return Task.CompletedTask; + + MessagingInstruments.Completed.Add(1, new KeyValuePair("source", _entry.Destination.Key)); + return _transport.CompleteAsync(_entry, cancellationToken); + } + + public async Task RejectAsync(RejectOptions? options = null, CancellationToken cancellationToken = default) + { + if (!TryMarkHandled()) + return; + + options ??= new RejectOptions(); + + if (options.Terminal) + { + MessagingInstruments.DeadLettered.Add(1, new KeyValuePair("source", _entry.Destination.Key)); + var enriched = _entry with { Headers = BuildDeadLetterHeaders(_entry, Attempts, options.Exception, _timeProvider) }; + await DeadLetterOrDropAsync(_transport, enriched, options.Reason, _deadLetterDestination, _logger, cancellationToken).AnyContext(); + return; + } + + MessagingInstruments.Abandoned.Add(1, new KeyValuePair("source", _entry.Destination.Key)); + + if (options.RedeliveryDelay is not { } redeliveryDelay || redeliveryDelay <= TimeSpan.Zero) + { + await _transport.AbandonAsync(_entry, cancellationToken).AnyContext(); + return; + } + + // Honor an explicit redelivery delay natively when the transport can (within its advertised maximum); otherwise + // re-schedule the message through the runtime store and complete the original so the delay survives transports + // without native delayed redelivery. + if (_transport is ISupportsRedeliveryDelay redelivery && (redelivery.MaxRedeliveryDelay is not { } max || redeliveryDelay <= max)) + { + await redelivery.AbandonAsync(_entry, redeliveryDelay, cancellationToken).AnyContext(); + return; + } + + // The runtime-store fallback re-sends the message as a plain queue send, which only makes sense for a + // queue-channel entry: a subscription-channel entry would need to be re-sent into its subscription group, and + // a queue send to that address would land where no subscription group reads. + bool isSubscriptionSource = _entry.Destination.Role == DestinationRole.Subscription; + if (_runtimeStore is null || isSubscriptionSource) + { + // A best-effort delay (the core retry policy) degrades to immediate redelivery; an explicit caller delay + // stays strict because the caller is depending on the timing. + if (options.BestEffortDelay) + { + await _transport.AbandonAsync(_entry, cancellationToken).AnyContext(); + return; + } + + throw new MessageBusException($"Delayed redelivery of \"{_entry.Destination.Key}\" requires native redelivery-delay support from transport \"{_transport.GetType().Name}\" (within its supported maximum){(isSubscriptionSource ? "" : " or a registered job runtime store")}."); + } + + // Advance from the reconciled attempt count, not the raw transport DeliveryCount: the re-send produces a new + // transport message whose native DeliveryCount resets to 1, so basing the next attempt on DeliveryCount would + // pin it at 2 and redeliver forever. Attempts already takes the max of DeliveryCount and the carried header. + int nextAttempt = Attempts + 1; + var headers = _entry.Headers.ToBuilder() + .Set(KnownHeaders.Attempts, nextAttempt.ToString(CultureInfo.InvariantCulture)) + .Build(); + + await _runtimeStore.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = $"{_entry.Id}:retry:{nextAttempt}", + Kind = ScheduledDispatchKind.QueueMessage, + Destination = _entry.Destination, + Body = _entry.Body, + Headers = headers, + Options = new TransportSendOptions { Priority = Priority }, + DueUtc = _timeProvider.GetUtcNow().Add(redeliveryDelay) + }, cancellationToken).AnyContext(); + + await _transport.CompleteAsync(_entry, cancellationToken).AnyContext(); + } + + public Task RenewLockAsync(TimeSpan? duration = null, CancellationToken cancellationToken = default) + { + return _transport is ISupportsLockRenewal lockRenewal + ? lockRenewal.RenewLockAsync(_entry, duration, cancellationToken) + : throw new NotSupportedException($"Transport \"{_transport.GetType().Name}\" does not support lock renewal."); + } + + // Terminal settlement. Prefer the transport's native dead-letter sink (preserves native DLQ tooling). When the + // transport has none, copy the raw entry to the configured dead-letter destination — or the derived + // "{source}.deadletter" when none is configured, so a dead message is always parked somewhere inspectable — + // recording the reason, then complete the original. Parking is best-effort: a park failure logs and drops rather + // than throwing, because a terminal settle must never stall the consumer loop. + internal static async Task DeadLetterOrDropAsync(IMessageTransport transport, TransportEntry entry, string? reason, string? deadLetterDestination, ILogger logger, CancellationToken cancellationToken) + { + if (transport is ISupportsDeadLetter deadLetter) + { + await deadLetter.DeadLetterAsync(entry, reason, cancellationToken).AnyContext(); + return; + } + + var destination = !String.IsNullOrEmpty(deadLetterDestination) + ? DestinationAddress.ForQueue(deadLetterDestination) + : DestinationAddress.ForQueue($"{entry.Destination.Key}.deadletter"); + var headers = String.IsNullOrEmpty(reason) + ? entry.Headers + : entry.Headers.ToBuilder().Set(KnownHeaders.DeadLetterReason, reason).Build(); + + try + { + await transport.SendAsync(destination, [new TransportMessage { Body = entry.Body, Headers = headers, MessageId = entry.Id }], new TransportSendOptions(), cancellationToken).AnyContext(); + } + catch (Exception ex) + { + logger.LogError(ex, "Failed to park dead-lettered message \"{MessageId}\" at \"{Destination}\"; dropping it: {Message}", entry.Id, destination.Key, ex.Message); + } + + await transport.CompleteAsync(entry, cancellationToken).AnyContext(); + } + + // Stamps the dead-letter forensics contract (see KnownHeaders) so a dead message is triageable — exception details, + // reconciled attempt count, where it was consumed from, and when it died. The attempt count goes in a forensics + // header (never message.attempts) so a replayed message starts with a fresh retry budget. + internal static MessageHeaders BuildDeadLetterHeaders(TransportEntry entry, int attempts, Exception? exception, TimeProvider timeProvider) + { + var headers = entry.Headers.ToBuilder() + .Set(KnownHeaders.DeadLetterAttempts, attempts.ToString(CultureInfo.InvariantCulture)) + .Set(KnownHeaders.DeadLetterFailedAt, timeProvider.GetUtcNow().ToString("O", CultureInfo.InvariantCulture)) + .Set(KnownHeaders.DeadLetterOriginalDestination, entry.Destination.Key); + + if (exception is not null) + { + headers.Set(KnownHeaders.DeadLetterExceptionType, exception.GetType().FullName ?? exception.GetType().Name); + headers.Set(KnownHeaders.DeadLetterExceptionMessage, Truncate(exception.Message, 1024)); + if (exception.StackTrace is { } stack) + headers.Set(KnownHeaders.DeadLetterExceptionStackTrace, Truncate(stack, 4096)); + } + else + { + // A death with no exception (no-handler, unresolved-type) must not carry stale forensics from a previous one. + headers.Remove(KnownHeaders.DeadLetterExceptionType); + headers.Remove(KnownHeaders.DeadLetterExceptionMessage); + headers.Remove(KnownHeaders.DeadLetterExceptionStackTrace); + } + + return headers.Build(); + } + + private static string Truncate(string value, int maxLength) + { + return value.Length <= maxLength ? value : value[..maxLength]; + } + + private bool TryMarkHandled() + { + return Interlocked.CompareExchange(ref _isHandled, 1, 0) == 0; + } + + private static int ParseAttemptsHeader(MessageHeaders headers) + { + return Int32.TryParse(headers.GetValueOrDefault(KnownHeaders.Attempts), NumberStyles.Integer, CultureInfo.InvariantCulture, out int attempts) && attempts > 0 + ? attempts + : 0; + } +} + +internal sealed class MessageContext : MessageContext, IMessageContext where T : class +{ + public MessageContext(IMessageTransport transport, TransportEntry entry, T message, CancellationToken cancellationToken, IScheduledDispatchStore? runtimeStore = null, TimeProvider? timeProvider = null, string? deadLetterDestination = null, ILogger? logger = null) + : base(transport, entry, cancellationToken, runtimeStore, timeProvider, deadLetterDestination, logger) + { + Message = message; + } + + public T Message { get; } +} + +internal static class MessageRoutingConventions +{ + public static string ToKebabCase(string value) + { + if (String.IsNullOrEmpty(value)) + return value; + + Span buffer = stackalloc char[value.Length * 2]; + int position = 0; + for (int index = 0; index < value.Length; index++) + { + char current = value[index]; + if (Char.IsUpper(current)) + { + if (index > 0) + buffer[position++] = '-'; + + buffer[position++] = Char.ToLowerInvariant(current); + } + else + { + buffer[position++] = current; + } + } + + return new String(buffer[..position]); + } +} + +/// +/// A started listener handle for one channel (a send destination or a topic subscription); the bus composes one per +/// channel into the it returns. +/// +internal sealed class MessageListenerHandle : IAsyncDisposable +{ + private readonly Func _dispose; + private int _isDisposed; + + public MessageListenerHandle(DestinationAddress source, string key, Func dispose) + { + Source = source; + Key = key; + _dispose = dispose; + } + + public DestinationAddress Source { get; } + public string Topic => Source.Topic ?? ""; + public string Subscription => Source.Role == DestinationRole.Subscription ? Source.Name : ""; + public string Key { get; } + + // Disposing a single consumer handle detaches just that consumer from its source listener; the underlying receive + // loop keeps running until its last consumer detaches. + public async ValueTask DisposeAsync() + { + if (Interlocked.Exchange(ref _isDisposed, 1) == 1) + return; + + await _dispose().AnyContext(); + } +} + +internal sealed record MessageListenerRegistration +{ + public required Type MessageType { get; init; } + public required DestinationAddress Source { get; init; } + public required Delegate Handler { get; init; } + public required AckMode AckMode { get; init; } + public required int MaxConcurrency { get; init; } + public required int? MaxAttempts { get; init; } + public required Func? RedeliveryBackoff { get; init; } + public required Func? DeadLetterWhen { get; init; } + + public static MessageListenerRegistration Create(Delegate handler, ListenerConfig config) + { + return new MessageListenerRegistration + { + MessageType = config.MessageType, + Source = config.Source, + Handler = handler, + AckMode = config.AckMode, + MaxConcurrency = Math.Max(1, config.MaxConcurrency), + MaxAttempts = config.MaxAttempts, + RedeliveryBackoff = config.RedeliveryBackoff, + DeadLetterWhen = config.DeadLetterWhen + }; + } + + public bool Matches(MessageListenerRegistration other) + { + // Failure policies are compared by delegate identity, not mere presence: subscriptions sharing a consumer Key + // form ONE competing group, and two members whose retry/dead-letter LOGIC differs would settle the same + // message differently depending on which member happened to receive it. Callers sharing a Key must share the + // actual delegate instances. + return MessageType == other.MessageType + && Source == other.Source + && Handler == other.Handler + && AckMode == other.AckMode + && MaxConcurrency == other.MaxConcurrency + && MaxAttempts == other.MaxAttempts + && Equals(RedeliveryBackoff, other.RedeliveryBackoff) + && Equals(DeadLetterWhen, other.DeadLetterWhen); + } +} diff --git a/src/Foundatio/Messaging/MessageHandlerHostedService.cs b/src/Foundatio/Messaging/MessageHandlerHostedService.cs new file mode 100644 index 000000000..627061b90 --- /dev/null +++ b/src/Foundatio/Messaging/MessageHandlerHostedService.cs @@ -0,0 +1,124 @@ +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Utility; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; + +namespace Foundatio.Messaging; + +/// +/// One declarative message-handler registration: a description for logging and a factory that starts the underlying +/// queue consumer or pub/sub subscription and returns it for disposal on shutdown. Built by the AddHandler +/// builder methods, which bind the message type at compile time (one registration per delivery verb). +/// +internal sealed class MessageHandlerRegistration +{ + public required string Description { get; init; } + public required Func> StartAsync { get; init; } +} + +/// The DI-selected , applied by the handler host at startup and by the message clients on use. +internal sealed record MessagingTopologyOptions(TopologyMode Mode); + +/// +/// Hosts every declaratively-registered message handler for the app's lifetime: on start it launches each handler's +/// consumer/subscription; on stop it disposes them. Auto-registered when the first handler is added, so users register +/// handlers in configuration and never hand-write a hosted service. Programmatic +/// remain available for dynamic use. +/// +internal sealed class MessageHandlerHostedService : IHostedService +{ + private readonly IServiceProvider _serviceProvider; + private readonly IEnumerable _registrations; + private readonly ILogger _logger; + private readonly List _started = new(); + + public MessageHandlerHostedService(IServiceProvider serviceProvider, IEnumerable registrations, ILoggerFactory? loggerFactory = null) + { + _serviceProvider = serviceProvider; + _registrations = registrations; + _logger = loggerFactory?.CreateLogger() ?? NullLogger.Instance; + } + + public async Task StartAsync(CancellationToken cancellationToken) + { + await ApplyTopologyAsync(cancellationToken).AnyContext(); + + try + { + foreach (var registration in _registrations) + { + var disposable = await registration.StartAsync(_serviceProvider, cancellationToken).AnyContext(); + _started.Add(disposable); + _logger.LogInformation("Started message handler {Handler}", registration.Description); + } + } + catch + { + // A hosted service whose StartAsync throws is not sent StopAsync, so dispose whatever we already started + // rather than leaking those consumers' background receive loops. + await DisposeStartedAsync().AnyContext(); + throw; + } + } + + // Apply the app's declared topology before any handler starts consuming: Ensure creates what the routing config + // declares, Validate proves it exists and fails startup when it doesn't (a missing destination should stop the app + // at boot, not surface as runtime send errors), and None trusts out-of-band provisioning entirely. + private async Task ApplyTopologyAsync(CancellationToken cancellationToken) + { + var mode = (_serviceProvider.GetService(typeof(MessagingTopologyOptions)) as MessagingTopologyOptions)?.Mode ?? TopologyMode.Ensure; + if (mode == TopologyMode.None) + return; + + if (_serviceProvider.GetService(typeof(IMessageTopology)) is not IMessageTopology topology) + return; + + if (mode == TopologyMode.Validate) + { + await topology.ValidateAsync(cancellationToken).AnyContext(); + _logger.LogInformation("Validated declared message topology"); + return; + } + + try + { + await topology.EnsureAsync(cancellationToken).AnyContext(); + _logger.LogInformation("Ensured declared message topology"); + } + catch (NotSupportedException) + { + // The transport cannot provision; the runtime use-time paths no-op the same way, so startup should not fail. + _logger.LogDebug("Transport does not support topology provisioning; skipping startup ensure"); + } + } + + public Task StopAsync(CancellationToken cancellationToken) => DisposeStartedAsync(); + + private async Task DisposeStartedAsync() + { + try + { + // Dispose every started consumer even if one throws (e.g. a broker connection dropped mid-shutdown), so a + // single failure can't leak the rest. + foreach (var disposable in _started) + { + try + { + await disposable.DisposeAsync().AnyContext(); + } + catch (Exception ex) + { + _logger.LogError(ex, "Error disposing message handler consumer: {Message}", ex.Message); + } + } + } + finally + { + _started.Clear(); + } + } +} diff --git a/src/Foundatio/Messaging/MessageHeaders.cs b/src/Foundatio/Messaging/MessageHeaders.cs new file mode 100644 index 000000000..a5e8ae045 --- /dev/null +++ b/src/Foundatio/Messaging/MessageHeaders.cs @@ -0,0 +1,142 @@ +using System; +using System.Collections; +using System.Collections.Frozen; +using System.Collections.Generic; +using System.Text.Json; + +namespace Foundatio.Messaging; + +public sealed class MessageHeaders : IReadOnlyDictionary +{ + public static MessageHeaders Empty { get; } = new(FrozenDictionary.Empty); + + private readonly FrozenDictionary _headers; + + private MessageHeaders(FrozenDictionary headers) + { + _headers = headers; + } + + public string this[string key] => _headers[key]; + public IEnumerable Keys => _headers.Keys; + public IEnumerable Values => _headers.Values; + public int Count => _headers.Count; + + public static MessageHeaders Create(IEnumerable> headers) + { + ArgumentNullException.ThrowIfNull(headers); + + if (headers is MessageHeaders messageHeaders) + return messageHeaders; + + var values = new Dictionary(StringComparer.OrdinalIgnoreCase); + foreach (var header in headers) + { + ArgumentException.ThrowIfNullOrEmpty(header.Key); + ArgumentNullException.ThrowIfNull(header.Value); + values[header.Key] = header.Value; + } + + return values.Count == 0 + ? Empty + : new MessageHeaders(values.ToFrozenDictionary(StringComparer.OrdinalIgnoreCase)); + } + + /// + /// The canonical on-the-wire encoding for headers (a JSON object). Transports should use this rather than rolling + /// their own so the round-trip semantics — notably case-insensitive keys (see ) — are identical + /// and contractually guaranteed across providers. + /// + public static string SerializeToJson(MessageHeaders headers) + { + ArgumentNullException.ThrowIfNull(headers); + var map = new Dictionary(StringComparer.Ordinal); + foreach (var header in headers) + map[header.Key] = header.Value; + return JsonSerializer.Serialize(map); + } + + /// Reads headers from the canonical encoding produced by . + public static MessageHeaders DeserializeFromJson(string? json) + { + if (String.IsNullOrEmpty(json)) + return Empty; + var map = JsonSerializer.Deserialize>(json); + return map is null ? Empty : Create(map); + } + + public bool ContainsKey(string key) + { + return _headers.ContainsKey(key); + } + + public bool TryGetValue(string key, out string value) + { + return _headers.TryGetValue(key, out value!); + } + + public string? GetValueOrDefault(string key) + { + return _headers.GetValueOrDefault(key); + } + + public Builder ToBuilder() + { + return new Builder(_headers); + } + + public IEnumerator> GetEnumerator() + { + return _headers.GetEnumerator(); + } + + IEnumerator IEnumerable.GetEnumerator() + { + return GetEnumerator(); + } + + public sealed class Builder + { + private readonly Dictionary _headers; + + internal Builder(IEnumerable> headers) + { + _headers = new Dictionary(headers, StringComparer.OrdinalIgnoreCase); + } + + public Builder Add(string key, string value) + { + ArgumentException.ThrowIfNullOrEmpty(key); + ArgumentNullException.ThrowIfNull(value); + _headers.Add(key, value); + return this; + } + + public Builder Set(string key, string value) + { + ArgumentException.ThrowIfNullOrEmpty(key); + ArgumentNullException.ThrowIfNull(value); + _headers[key] = value; + return this; + } + + public Builder SetIfMissing(string key, string value) + { + ArgumentException.ThrowIfNullOrEmpty(key); + ArgumentNullException.ThrowIfNull(value); + _headers.TryAdd(key, value); + return this; + } + + public bool Remove(string key) + { + ArgumentException.ThrowIfNullOrEmpty(key); + return _headers.Remove(key); + } + + public MessageHeaders Build() + { + return Create(_headers); + } + } +} diff --git a/src/Foundatio/Messaging/MessageRouteAttribute.cs b/src/Foundatio/Messaging/MessageRouteAttribute.cs new file mode 100644 index 000000000..418816365 --- /dev/null +++ b/src/Foundatio/Messaging/MessageRouteAttribute.cs @@ -0,0 +1,21 @@ +using System; + +namespace Foundatio.Messaging; + +[AttributeUsage(AttributeTargets.Class | AttributeTargets.Struct, Inherited = false)] +public sealed class MessageRouteAttribute : Attribute +{ + public MessageRouteAttribute() + { + } + + public MessageRouteAttribute(string name) + { + Destination = name; + Topic = name; + } + + public string? Destination { get; set; } + public string? Topic { get; set; } + public string? Subscription { get; set; } +} diff --git a/src/Foundatio/Messaging/MessageRouting.cs b/src/Foundatio/Messaging/MessageRouting.cs new file mode 100644 index 000000000..6b7edf1b3 --- /dev/null +++ b/src/Foundatio/Messaging/MessageRouting.cs @@ -0,0 +1,317 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Reflection; + +namespace Foundatio.Messaging; + +public enum MessageRouteRole +{ + QueueDestination, + PubSubTopic +} + +public sealed record MessageRouteContext +{ + public required Type MessageType { get; init; } + public required MessageRouteRole Role { get; init; } + public string? OperationOverride { get; init; } +} + +public sealed record MessageSubscriptionContext +{ + public required Type MessageType { get; init; } + public required string Topic { get; init; } + public string? OperationOverride { get; init; } +} + +public interface IMessageRouter +{ + string ResolveRoute(MessageRouteContext context); + string ResolveSubscription(MessageSubscriptionContext context); +} + +public sealed record MessageRouteMap +{ + public required Type MessageType { get; init; } + public required MessageRouteRole Role { get; init; } + public required string Route { get; init; } +} + +public sealed class MessageRoutingOptions +{ + internal List RouteMaps { get; } = []; + internal List TopologyDeclarations { get; } = []; + + public string? DefaultQueueDestination { get; set; } + public string? DefaultPubSubTopic { get; set; } + public string? SubscriptionIdentity { get; set; } + public string? ServiceIdentity { get; set; } + public Func? Convention { get; set; } + + public IReadOnlyList GetTopologyDeclarations() + { + return TopologyDeclarations.ToArray(); + } + + internal void Declare(DestinationDeclaration declaration) + { + ArgumentNullException.ThrowIfNull(declaration); + + if (!TopologyDeclarations.Any(d => d.Address == declaration.Address)) + TopologyDeclarations.Add(declaration); + } + + internal void RemoveDeclarations(Predicate match) + { + TopologyDeclarations.RemoveAll(match); + } +} + +public sealed class MessageRoutingOptionsBuilder +{ + private readonly MessageRoutingOptions _options; + + public MessageRoutingOptionsBuilder() + : this(new MessageRoutingOptions()) + { + } + + internal MessageRoutingOptionsBuilder(MessageRoutingOptions options) + { + _options = options; + } + + public MessageRoutingOptionsBuilder UseDefaultQueue(string destination) + { + ArgumentException.ThrowIfNullOrEmpty(destination); + _options.DefaultQueueDestination = destination; + DeclareQueue(destination); + return this; + } + + public MessageRoutingOptionsBuilder UseDefaultTopic(string topic) + { + ArgumentException.ThrowIfNullOrEmpty(topic); + _options.DefaultPubSubTopic = topic; + DeclareTopic(topic); + return this; + } + + public MessageRoutingOptionsBuilder MapQueue(string destination) + { + return MapQueue(typeof(T), destination); + } + + public MessageRoutingOptionsBuilder MapQueue(Type messageType, string destination) + { + return Map(MessageRouteRole.QueueDestination, destination, messageType); + } + + public MessageRoutingOptionsBuilder MapQueue(string destination, params Type[] messageTypes) + { + return Map(MessageRouteRole.QueueDestination, destination, messageTypes); + } + + public MessageRoutingOptionsBuilder MapTopic(string topic) + { + return MapTopic(typeof(T), topic); + } + + public MessageRoutingOptionsBuilder MapTopic(Type messageType, string topic) + { + return Map(MessageRouteRole.PubSubTopic, topic, messageType); + } + + public MessageRoutingOptionsBuilder MapTopic(string topic, params Type[] messageTypes) + { + return Map(MessageRouteRole.PubSubTopic, topic, messageTypes); + } + + public MessageRoutingOptionsBuilder UseSubscriptionIdentity(string subscription) + { + ArgumentException.ThrowIfNullOrEmpty(subscription); + _options.SubscriptionIdentity = subscription; + RebuildSubscriptionDeclarations(); + return this; + } + + public MessageRoutingOptionsBuilder UseServiceIdentity(string serviceIdentity) + { + ArgumentException.ThrowIfNullOrEmpty(serviceIdentity); + _options.ServiceIdentity = serviceIdentity; + RebuildSubscriptionDeclarations(); + return this; + } + + public MessageRoutingOptionsBuilder UseConvention(Func convention) + { + _options.Convention = convention ?? throw new ArgumentNullException(nameof(convention)); + return this; + } + + + public MessageRoutingOptions Build() + { + return _options; + } + + private MessageRoutingOptionsBuilder Map(MessageRouteRole role, string route, params Type[] messageTypes) + { + ArgumentException.ThrowIfNullOrEmpty(route); + ArgumentNullException.ThrowIfNull(messageTypes); + + if (messageTypes.Length == 0) + throw new ArgumentException("At least one message type is required.", nameof(messageTypes)); + + foreach (var messageType in messageTypes) + { + ArgumentNullException.ThrowIfNull(messageType); + _options.RouteMaps.Add(new MessageRouteMap + { + MessageType = messageType, + Role = role, + Route = route + }); + } + + if (role == MessageRouteRole.QueueDestination) + DeclareQueue(route); + else + DeclareTopic(route); + + return this; + } + + private void DeclareQueue(string destination) + { + _options.Declare(new DestinationDeclaration { Address = DestinationAddress.ForQueue(destination) }); + } + + private void DeclareTopic(string topic) + { + _options.Declare(new DestinationDeclaration { Address = DestinationAddress.ForTopic(topic) }); + DeclareSubscription(topic); + } + + private void RebuildSubscriptionDeclarations() + { + _options.RemoveDeclarations(d => d.Address.Role == DestinationRole.Subscription); + + if (!String.IsNullOrEmpty(_options.DefaultPubSubTopic)) + DeclareSubscription(_options.DefaultPubSubTopic); + + foreach (string topic in _options.RouteMaps + .Where(m => m.Role == MessageRouteRole.PubSubTopic) + .Select(m => m.Route) + .Distinct(StringComparer.Ordinal)) + { + DeclareSubscription(topic); + } + } + + private void DeclareSubscription(string topic) + { + string? subscription = _options.SubscriptionIdentity ?? _options.ServiceIdentity; + if (String.IsNullOrEmpty(subscription)) + return; + + DeclareSubscription(topic, subscription); + } + + private void DeclareSubscription(string topic, string subscription) + { + // The SAME canonical address the runtime subscribe path ensures and receives from — declaring the bare + // subscription name here while the runtime used a topic-qualified string is exactly the topology-vs-runtime + // identity mismatch DestinationAddress exists to prevent. + _options.Declare(new DestinationDeclaration { Address = DestinationAddress.ForSubscription(topic, subscription) }); + } +} + +public sealed class DefaultMessageRouter : IMessageRouter +{ + public static DefaultMessageRouter Instance { get; } = new(new MessageRoutingOptions()); + + private readonly MessageRoutingOptions _options; + + public DefaultMessageRouter(MessageRoutingOptions options) + { + _options = options ?? throw new ArgumentNullException(nameof(options)); + } + + public string ResolveRoute(MessageRouteContext context) + { + ArgumentNullException.ThrowIfNull(context); + ArgumentNullException.ThrowIfNull(context.MessageType); + + if (!String.IsNullOrEmpty(context.OperationOverride)) + return context.OperationOverride; + + var exact = _options.RouteMaps.LastOrDefault(m => m.Role == context.Role && m.MessageType == context.MessageType); + if (exact is not null) + return exact.Route; + + var assignable = _options.RouteMaps.LastOrDefault(m => m.Role == context.Role && m.MessageType != context.MessageType && m.MessageType.IsAssignableFrom(context.MessageType)); + if (assignable is not null) + return assignable.Route; + + var attribute = context.MessageType.GetCustomAttribute(); + string? attributedRoute = context.Role == MessageRouteRole.QueueDestination + ? attribute?.Destination + : attribute?.Topic ?? attribute?.Destination; + + if (!String.IsNullOrEmpty(attributedRoute)) + return attributedRoute; + + string? configuredDefault = context.Role == MessageRouteRole.QueueDestination + ? _options.DefaultQueueDestination + : _options.DefaultPubSubTopic; + + if (!String.IsNullOrEmpty(configuredDefault)) + return configuredDefault; + + if (_options.Convention is not null) + { + string convention = _options.Convention(context); + if (!String.IsNullOrEmpty(convention)) + return convention; + } + + return MessageRoutingConventions.ToKebabCase(context.MessageType.Name); + } + + public string ResolveSubscription(MessageSubscriptionContext context) + { + ArgumentNullException.ThrowIfNull(context); + ArgumentNullException.ThrowIfNull(context.MessageType); + ArgumentException.ThrowIfNullOrEmpty(context.Topic); + + if (!String.IsNullOrEmpty(context.OperationOverride)) + return context.OperationOverride; + + if (!String.IsNullOrEmpty(_options.SubscriptionIdentity)) + return _options.SubscriptionIdentity; + + if (context.MessageType.GetCustomAttribute()?.Subscription is { Length: > 0 } subscription) + return subscription; + + if (!String.IsNullOrEmpty(_options.ServiceIdentity)) + return _options.ServiceIdentity; + + return GetDefaultServiceIdentity(); + } + + + private static string GetDefaultServiceIdentity() + { + string? configured = Environment.GetEnvironmentVariable("FOUNDATIO_SUBSCRIPTION_ID"); + if (!String.IsNullOrEmpty(configured)) + return configured; + + configured = Environment.GetEnvironmentVariable("FOUNDATIO_SERVICE_ID"); + if (!String.IsNullOrEmpty(configured)) + return configured; + + return MessageRoutingConventions.ToKebabCase(AppDomain.CurrentDomain.FriendlyName); + } +} diff --git a/src/Foundatio/Messaging/MessageTopology.cs b/src/Foundatio/Messaging/MessageTopology.cs new file mode 100644 index 000000000..b5ad04702 --- /dev/null +++ b/src/Foundatio/Messaging/MessageTopology.cs @@ -0,0 +1,64 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Utility; + +namespace Foundatio.Messaging; + +public interface IMessageTopology +{ + IReadOnlyList GetDeclarations(); + Task EnsureAsync(CancellationToken cancellationToken = default); + Task ValidateAsync(CancellationToken cancellationToken = default); +} + +public sealed class MessageTopology : IMessageTopology +{ + private readonly IMessageTransport _transport; + private readonly MessageRoutingOptions _options; + + public MessageTopology(IMessageTransport transport, MessageRoutingOptions options) + { + _transport = transport ?? throw new ArgumentNullException(nameof(transport)); + _options = options ?? throw new ArgumentNullException(nameof(options)); + } + + public IReadOnlyList GetDeclarations() + { + return _options.GetTopologyDeclarations(); + } + + public async Task EnsureAsync(CancellationToken cancellationToken = default) + { + var declarations = GetDeclarations(); + if (declarations.Count == 0) + return; + + if (_transport is not ISupportsProvisioning provisioning) + throw new NotSupportedException($"Transport \"{_transport.GetType().Name}\" does not support topology provisioning."); + + await provisioning.EnsureAsync(declarations, cancellationToken).AnyContext(); + } + + public async Task ValidateAsync(CancellationToken cancellationToken = default) + { + var declarations = GetDeclarations(); + if (declarations.Count == 0) + return; + + if (_transport is not ISupportsProvisioning provisioning) + throw new NotSupportedException($"Transport \"{_transport.GetType().Name}\" does not support topology validation."); + + var missing = new List(); + foreach (var declaration in declarations) + { + if (!await provisioning.ExistsAsync(declaration.Address, cancellationToken).AnyContext()) + missing.Add(declaration); + } + + if (missing.Count > 0) + throw new InvalidOperationException($"Message topology is missing: {String.Join(", ", missing.Select(d => d.Address.ToString()))}."); + } +} diff --git a/src/Foundatio/Messaging/MessageTransport.cs b/src/Foundatio/Messaging/MessageTransport.cs new file mode 100644 index 000000000..19bac2841 --- /dev/null +++ b/src/Foundatio/Messaging/MessageTransport.cs @@ -0,0 +1,308 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; + +namespace Foundatio.Messaging; + +public enum MessagePriority +{ + Low = 0, + Normal = 1, + High = 2 +} + +public enum DeliveryGuarantee +{ + AtMostOnce, + AtLeastOnce +} + +public enum OrderingGuarantee +{ + None, + Fifo, + PerPartition +} + +public enum DestinationRole +{ + Queue, + Topic, + Subscription, + Binding +} + +/// +/// The canonical identity of a transport destination: a name, the role that names the physical namespace it lives in, +/// and — for subscriptions — the owning topic. Every transport API (send, receive, subscribe, stats, settlement, +/// provisioning) uses this one value, so the same logical destination can never be spelled two ways on two paths. +/// +/// +/// is the destination's opaque string form ("{topic}/{name}" for subscriptions, Name +/// otherwise) for logging, metrics tags, and dictionary keys. Because a subscription key contains '/', a +/// transport must NOT assume it is a legal broker resource name (e.g. an SQS queue name) — map it to native resources +/// during and treat it as a lookup key thereafter. Topic and +/// subscription names must not contain '/'. +/// +public sealed record DestinationAddress +{ + public required string Name { get; init; } + public DestinationRole Role { get; init; } = DestinationRole.Queue; + + /// The owning topic when is ; null otherwise. + public string? Topic { get; init; } + + /// The canonical opaque string form: "{topic}/{name}" for subscriptions, Name otherwise. + public string Key => Topic is { Length: > 0 } topic ? $"{topic}/{Name}" : Name; + + public override string ToString() => $"{Role}:{Key}"; + + public static DestinationAddress ForQueue(string name) + { + ArgumentException.ThrowIfNullOrEmpty(name); + return new DestinationAddress { Name = name, Role = DestinationRole.Queue }; + } + + public static DestinationAddress ForTopic(string name) + { + ArgumentException.ThrowIfNullOrEmpty(name); + return new DestinationAddress { Name = name, Role = DestinationRole.Topic }; + } + + public static DestinationAddress ForSubscription(string topic, string subscription) + { + ArgumentException.ThrowIfNullOrEmpty(topic); + ArgumentException.ThrowIfNullOrEmpty(subscription); + return new DestinationAddress { Name = subscription, Role = DestinationRole.Subscription, Topic = topic }; + } +} + +public sealed record TransportMessage +{ + public required ReadOnlyMemory Body { get; init; } + public MessageHeaders Headers { get; init; } = MessageHeaders.Empty; + public string? MessageId { get; init; } + + /// + /// Content type of (e.g. application/json). A transport whose native wire format is text + /// (such as SQS/SNS) can store a text body directly when this indicates text, avoiding base64 overhead; null means + /// unknown, so a byte-safe encoding should be used. + /// + public string? ContentType { get; init; } +} + +public sealed record TransportSendOptions +{ + public MessagePriority Priority { get; init; } = MessagePriority.Normal; + public DateTimeOffset? DeliverAt { get; init; } +} + +public sealed record TransportEntry +{ + public required string Id { get; init; } + public required DestinationAddress Destination { get; init; } + public required ReadOnlyMemory Body { get; init; } + public MessageHeaders Headers { get; init; } = MessageHeaders.Empty; + public int DeliveryCount { get; init; } = 1; + public DateTimeOffset? EnqueuedUtc { get; init; } + public required Receipt Receipt { get; init; } +} + +public readonly struct Receipt +{ + public object? TransportState { get; init; } +} + +public sealed record ReceiveRequest +{ + public int MaxMessages { get; init; } = 1; + public TimeSpan? MaxWaitTime { get; init; } +} + +public sealed record MessageDestinationStats +{ + // Point-in-time gauges every transport can report (may be approximate / eventually consistent on real brokers, + // e.g. SQS ApproximateNumberOf*). + public long Queued { get; init; } + public long Working { get; init; } + public long Deadletter { get; init; } + + // Lifetime counters. Not universally available — a transport that does not track a counter leaves it null (e.g. + // SQS exposes no lifetime "completed" count). Null means "not reported", distinct from a reported zero. + public long? Enqueued { get; init; } + public long? Dequeued { get; init; } + public long? Completed { get; init; } + public long? Abandoned { get; init; } + public long? Errors { get; init; } + public long? Timeouts { get; init; } +} + +public sealed record SendItemResult +{ + /// The broker-assigned id of the accepted message. + public string? MessageId { get; init; } +} + +/// +/// The result of a successful : the accepted messages' ids, in order. +/// +/// +/// Send is throw-on-failure: a transport throws for any failure rather than returning a failed item, so every item in +/// was accepted. A multi-message send is NOT atomic — if a later message fails, earlier messages +/// may already have been delivered before the exception propagates. +/// +public sealed record SendResult +{ + public required IReadOnlyList Items { get; init; } +} + +/// +/// Thrown when a transport settle operation is given a receipt that has expired or was already settled. Strict receipt +/// validation is transport-specific: some brokers (e.g. SQS) treat settling with a stale receipt as idempotent and do +/// not raise, so callers must not depend on this exception for correctness — it is a best-effort safety signal. +/// +public sealed class ReceiptExpiredException : Exception +{ + public ReceiptExpiredException() : base("The transport receipt has expired or has already been settled.") { } + + public ReceiptExpiredException(string message) : base(message) { } + + public ReceiptExpiredException(string message, Exception innerException) : base(message, innerException) { } +} + +public sealed record DestinationDeclaration +{ + /// The canonical identity of the destination to provision — the SAME address the runtime later sends to, + /// receives from, and asks stats for, so provisioning and runtime can never disagree on a destination's identity. + public required DestinationAddress Address { get; init; } + + // Provider-specific creation arguments for transports that provision destinations (e.g. RabbitMQ queue arguments). + // Retry and dead-letter behavior is owned by the core RetryPolicy, not declared here, so destinations stay simple. + public IReadOnlyDictionary? ProviderArguments { get; init; } +} + +public sealed record PushOptions +{ + public int MaxConcurrentMessages { get; init; } = 1; + public TimeSpan PollInterval { get; init; } = TimeSpan.FromSeconds(1); +} + +/// +/// The capability and limit facts a transport advertises for one . Capabilities vary by +/// role on real brokers (SQS queues take DelaySeconds; SNS topics have no native delay), so the core asks per role +/// via rather than reading transport-wide flags. Anything not advertised +/// here is treated as unsupported: the core validates, falls back, or throws instead of letting the broker silently +/// drop a requested behavior. +/// +public sealed record TransportCapabilities +{ + /// Capabilities of a transport (or role) that advertises nothing: every feature routes through core fallbacks or fails validation. + public static readonly TransportCapabilities None = new(); + + /// The destination honors natively. + public bool DelayedDelivery { get; init; } + + /// + /// The longest delivery delay honored natively when is true (e.g. SQS caps + /// DelaySeconds at 15 minutes); null means unbounded. A send scheduled further out is routed through the + /// runtime-store fallback instead of being silently truncated to the broker's ceiling. + /// + public TimeSpan? MaxDeliveryDelay { get; init; } + + /// The destination honors . + public bool Priority { get; init; } + + /// The destination honors per-message expiration (). + public bool Expiration { get; init; } + + public OrderingGuarantee Ordering { get; init; } = OrderingGuarantee.None; + + /// Maximum messages per call; null means unbounded. The core chunks larger sends. + public int? MaxBatchSize { get; init; } + + /// Maximum message body size in bytes; null means unbounded. The core rejects oversized messages up front. + public long? MaxMessageBytes { get; init; } +} + +public interface ITransportInfo +{ + DeliveryGuarantee DeliveryGuarantee { get; } + IReadOnlySet SupportedRoles { get; } + + /// + /// The capabilities and limits this transport honors for destinations of the given role. Must be side-effect free + /// and cheap; the core consults it on every send-path decision (native delay vs. runtime-store fallback, + /// priority/expiration validation, size and batch limits). + /// + TransportCapabilities GetCapabilities(DestinationRole role); +} + +public interface IMessageTransport : IAsyncDisposable +{ + Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default); + Task CompleteAsync(TransportEntry entry, CancellationToken ct = default); + Task AbandonAsync(TransportEntry entry, CancellationToken ct = default); +} + +public interface ISupportsPull : IMessageTransport +{ + Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct = default); +} + +public interface ISupportsPush : IMessageTransport +{ + Task SubscribeAsync(DestinationAddress source, Func onMessage, PushOptions options, CancellationToken ct = default); +} + +public interface ISupportsRedeliveryDelay : IMessageTransport +{ + // The longest redelivery delay the transport can honor natively (e.g. SQS serves this via ChangeMessageVisibility, + // capped at 12 hours). Null means unbounded. A requested delay longer than this is routed through the runtime-store + // fallback instead of being silently clamped by the broker. + TimeSpan? MaxRedeliveryDelay { get; } + + Task AbandonAsync(TransportEntry entry, TimeSpan redeliveryDelay, CancellationToken ct = default); +} + +public interface ISupportsDeadLetter : IMessageTransport +{ + Task DeadLetterAsync(TransportEntry entry, string? reason, CancellationToken ct = default); + + // Reads dead-lettered entries for a destination so callers can inspect raw payloads (including poison messages + // that never deserialized) and the dead-letter reason header. Read entries are removed from the dead-letter store. + Task> ReceiveDeadLetteredAsync(DestinationAddress destination, ReceiveRequest request, CancellationToken ct = default); +} + +public interface ISupportsLockRenewal : IMessageTransport +{ + Task RenewLockAsync(TransportEntry entry, TimeSpan? duration, CancellationToken ct = default); +} + +public interface ISupportsVisibilityTimeout : IMessageTransport +{ + // The longest receive visibility timeout the transport can honor natively (e.g. SQS caps visibility at 12 hours). + // Null means unbounded. Callers requesting a longer visibility than the broker supports should treat that as + // unsatisfiable rather than relying on a silently clamped value. + TimeSpan? MaxVisibilityTimeout { get; } + + Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, TimeSpan visibility, CancellationToken ct = default); +} + +public interface ISupportsStats : IMessageTransport +{ + Task GetStatsAsync(DestinationAddress destination, CancellationToken ct = default); +} + +public interface ISupportsProvisioning : IMessageTransport +{ + Task EnsureAsync(IReadOnlyList declarations, CancellationToken ct = default); + Task DeleteAsync(DestinationAddress destination, CancellationToken ct = default); + Task ExistsAsync(DestinationAddress destination, CancellationToken ct = default); +} + +public interface IPushSubscription : IAsyncDisposable +{ + DestinationAddress Source { get; } +} diff --git a/src/Foundatio/Messaging/MessageTypeRegistry.cs b/src/Foundatio/Messaging/MessageTypeRegistry.cs new file mode 100644 index 000000000..6a1d80492 --- /dev/null +++ b/src/Foundatio/Messaging/MessageTypeRegistry.cs @@ -0,0 +1,72 @@ +using System; +using System.Collections.Generic; + +namespace Foundatio.Messaging; + +/// +/// Resolves the stable wire discriminator written to the message.type header in both directions: a CLR type to +/// its name (for sending) and a name back to its CLR type (so a grouped/interface consumer can deserialize the actual +/// payload type). Register stable names for types that may move between assemblies or namespaces; unregistered types +/// fall back to (never AssemblyQualifiedName). +/// +public interface IMessageTypeRegistry +{ + string GetName(Type messageType); + Type? Resolve(string name); +} + +public sealed record MessageTypeRegistration(string Name, Type MessageType); + +public sealed class MessageTypeRegistry : IMessageTypeRegistry +{ + private readonly Dictionary _nameToType = new(StringComparer.Ordinal); + private readonly Dictionary _typeToName = []; + + public MessageTypeRegistry(IEnumerable? registrations = null) + { + foreach (var registration in registrations ?? []) + Add(registration); + } + + public string GetName(Type messageType) + { + ArgumentNullException.ThrowIfNull(messageType); + return _typeToName.TryGetValue(messageType, out string? name) + ? name + : messageType.FullName ?? messageType.Name; + } + + public Type? Resolve(string name) + { + ArgumentException.ThrowIfNullOrEmpty(name); + + if (_nameToType.TryGetValue(name, out var registered)) + return registered; + + var type = Type.GetType(name, throwOnError: false); + if (type is not null) + return type; + + foreach (var assembly in AppDomain.CurrentDomain.GetAssemblies()) + { + type = assembly.GetType(name, throwOnError: false); + if (type is not null) + return type; + } + + return null; + } + + private void Add(MessageTypeRegistration registration) + { + ArgumentNullException.ThrowIfNull(registration); + ArgumentException.ThrowIfNullOrEmpty(registration.Name); + ArgumentNullException.ThrowIfNull(registration.MessageType); + + if (_nameToType.TryGetValue(registration.Name, out var existing) && existing != registration.MessageType) + throw new InvalidOperationException($"Message type name \"{registration.Name}\" is already registered for \"{existing.FullName}\"."); + + _nameToType[registration.Name] = registration.MessageType; + _typeToName[registration.MessageType] = registration.Name; + } +} diff --git a/src/Foundatio/Messaging/NullMessageBus.cs b/src/Foundatio/Messaging/NullMessageBus.cs index cee2fa42a..2eeb6dba6 100644 --- a/src/Foundatio/Messaging/NullMessageBus.cs +++ b/src/Foundatio/Messaging/NullMessageBus.cs @@ -2,7 +2,7 @@ using System.Threading; using System.Threading.Tasks; -namespace Foundatio.Messaging; +namespace Foundatio.Messaging.Legacy; public class NullMessageBus : IMessageBus { diff --git a/src/Foundatio/Messaging/SharedMessageBusOptions.cs b/src/Foundatio/Messaging/SharedMessageBusOptions.cs index 94a15423c..5f3bcc499 100644 --- a/src/Foundatio/Messaging/SharedMessageBusOptions.cs +++ b/src/Foundatio/Messaging/SharedMessageBusOptions.cs @@ -2,7 +2,7 @@ using System.Collections.Generic; using System.Diagnostics.CodeAnalysis; -namespace Foundatio.Messaging; +namespace Foundatio.Messaging.Legacy; public class SharedMessageBusOptions : SharedOptions { diff --git a/tests/Foundatio.Aws.Tests/AwsMessageTransportConformanceTests.cs b/tests/Foundatio.Aws.Tests/AwsMessageTransportConformanceTests.cs new file mode 100644 index 000000000..2f84d0227 --- /dev/null +++ b/tests/Foundatio.Aws.Tests/AwsMessageTransportConformanceTests.cs @@ -0,0 +1,44 @@ +using System; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Foundatio.Tests.Messaging; +using Xunit; + +namespace Foundatio.Aws.Tests; + +/// +/// Runs the shared transport conformance suite against AWS SQS/SNS. Set the environment variable +/// FOUNDATIO_AWS_CONNECTION_STRING (e.g. serviceurl=http://localhost:4566;accesskey=test;secretkey=test;region=us-east-1 +/// for LocalStack, or real AWS credentials) to run; when it is not set every test is skipped. Inherits every base +/// [Fact], so a new conformance check runs against SQS/SNS automatically; capabilities SQS/SNS do not support +/// (priority, per-message expiration, push delivery, transport-native dead-letter) self-skip via their +/// ISupports* capability checks in the base suite. +/// +public class AwsMessageTransportConformanceTests : MessageTransportConformanceTests +{ + // One prefix per test run isolates these queues/topics from prior runs and other environments on the same broker. + private static readonly string RunPrefix = "fnd-conf-" + Guid.NewGuid().ToString("N")[..8] + "-"; + + public AwsMessageTransportConformanceTests(ITestOutputHelper output) : base(output) { } + + protected override IMessageTransport? CreateTransport() + { + string? connectionString = Environment.GetEnvironmentVariable("FOUNDATIO_AWS_CONNECTION_STRING"); + if (String.IsNullOrEmpty(connectionString)) + return null; // not configured -> the base suite skips every test + + var options = AwsMessageTransportOptions.FromConnectionString(connectionString); + options.ResourcePrefix = RunPrefix; + return new AwsMessageTransport(options); + } + + [Fact] + public override Task CompleteAsync_WithExpiredReceipt_ThrowsReceiptExpiredExceptionAsync() + { + // Explicit, visible opt-out (not a silent skip): SQS DeleteMessage with a stale/used receipt handle is + // idempotent and does not raise. Strict receipt validation is transport-specific, not part of the shared + // contract, so SQS does not satisfy this check. + Assert.Skip("SQS DeleteMessage is idempotent for a stale receipt handle; strict receipt validation is not part of the shared contract."); + return Task.CompletedTask; + } +} diff --git a/tests/Foundatio.Aws.Tests/AwsMessageTransportTests.cs b/tests/Foundatio.Aws.Tests/AwsMessageTransportTests.cs new file mode 100644 index 000000000..99ca3f20d --- /dev/null +++ b/tests/Foundatio.Aws.Tests/AwsMessageTransportTests.cs @@ -0,0 +1,71 @@ +using System; +using System.Text; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Xunit; + +namespace Foundatio.Aws.Tests; + +public class AwsMessageTransportTests +{ + private static AwsMessageTransport? CreateTransport(string testName) + { + string? connectionString = Environment.GetEnvironmentVariable("FOUNDATIO_AWS_CONNECTION_STRING"); + if (String.IsNullOrEmpty(connectionString)) + return null; + + var options = AwsMessageTransportOptions.FromConnectionString(connectionString); + options.ResourcePrefix = $"fnd-{testName}-{Guid.NewGuid():N}"[..24] + "-"; + return new AwsMessageTransport(options); + } + + [Fact] + public async Task TextContentBody_RoundTripsThroughSqsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = CreateTransport("text"); + if (transport is null) + { + Assert.Skip("FOUNDATIO_AWS_CONNECTION_STRING not set."); + return; + } + + // Non-ASCII JSON exercises UTF-8 round-trip through the SQS string body (the text-content path that avoids base64). + string json = "{\"greeting\":\"héllo wörld\",\"n\":42}"; + await transport.EnsureAsync([new DestinationDeclaration { Address = DestinationAddress.ForQueue("text-body") }], cancellationToken); + + await transport.SendAsync(DestinationAddress.ForQueue("text-body"), + [new TransportMessage { Body = Encoding.UTF8.GetBytes(json), ContentType = "application/json" }], + new TransportSendOptions(), cancellationToken); + + var entries = await transport.ReceiveAsync(DestinationAddress.ForQueue("text-body"), new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, cancellationToken); + var entry = Assert.Single(entries); + Assert.Equal(json, Encoding.UTF8.GetString(entry.Body.Span)); + await transport.CompleteAsync(entry, cancellationToken); + } + + [Fact] + public async Task BinaryContentBody_RoundTripsThroughSqsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = CreateTransport("binary"); + if (transport is null) + { + Assert.Skip("FOUNDATIO_AWS_CONNECTION_STRING not set."); + return; + } + + // Non-UTF-8 bytes must still round-trip (via base64) when no text content type is declared. + byte[] payload = [0x00, 0x01, 0xFF, 0xFE, 0x10, 0x80]; + await transport.EnsureAsync([new DestinationDeclaration { Address = DestinationAddress.ForQueue("binary-body") }], cancellationToken); + + await transport.SendAsync(DestinationAddress.ForQueue("binary-body"), + [new TransportMessage { Body = payload }], + new TransportSendOptions(), cancellationToken); + + var entries = await transport.ReceiveAsync(DestinationAddress.ForQueue("binary-body"), new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, cancellationToken); + var entry = Assert.Single(entries); + Assert.Equal(payload, entry.Body.ToArray()); + await transport.CompleteAsync(entry, cancellationToken); + } +} diff --git a/tests/Foundatio.Aws.Tests/Foundatio.Aws.Tests.csproj b/tests/Foundatio.Aws.Tests/Foundatio.Aws.Tests.csproj new file mode 100644 index 000000000..d6d9f3e74 --- /dev/null +++ b/tests/Foundatio.Aws.Tests/Foundatio.Aws.Tests.csproj @@ -0,0 +1,10 @@ + + + + $(TestingPlatformCommandLineArguments) --ignore-exit-code 8 + + + + + + diff --git a/tests/Foundatio.Aws.Tests/README.md b/tests/Foundatio.Aws.Tests/README.md new file mode 100644 index 000000000..8828961cb --- /dev/null +++ b/tests/Foundatio.Aws.Tests/README.md @@ -0,0 +1,33 @@ +# Foundatio.Aws.Tests + +Runs the shared transport conformance suite (`MessageTransportConformanceTests`) against the AWS SQS/SNS +`IMessageTransport` (`Foundatio.Aws`). This is a temporary in-repo provider used to validate the redesigned transport +contract against a real broker before it is extracted to its own package. + +## Run against LocalStack + +```sh +# 1. Start LocalStack (SQS + SNS) +docker compose -f tests/Foundatio.Aws.Tests/docker-compose.yml up -d + +# 2. Point the tests at it +export FOUNDATIO_AWS_CONNECTION_STRING="serviceurl=http://localhost:4566;accesskey=test;secretkey=test;region=us-east-1" + +# 3. Run the conformance suite +dotnet test tests/Foundatio.Aws.Tests/Foundatio.Aws.Tests.csproj +``` + +When `FOUNDATIO_AWS_CONNECTION_STRING` is **not** set, every test is skipped (so the project is safe in CI without a broker). + +To run against real AWS, set the connection string to real credentials/region (omit `serviceurl`), e.g. +`accesskey=...;secretkey=...;region=us-east-1`. + +## Capability coverage + +SQS/SNS supports pull receive, visibility timeout, lock renewal, redelivery delay (12h cap), delayed delivery (15-min +cap), provisioning, and stats. It does **not** support per-message priority, per-message TTL/expiration, push delivery, +or transport-native dead-lettering (the core owns retry/dead-lettering). Conformance tests for those capabilities skip +automatically via their `ISupports*` checks. + +Each run uses a unique `ResourcePrefix` so leftover messages from a prior run cannot leak in. LocalStack state is +ephemeral; restart the container to reset. diff --git a/tests/Foundatio.Aws.Tests/docker-compose.yml b/tests/Foundatio.Aws.Tests/docker-compose.yml new file mode 100644 index 000000000..053c8bc0d --- /dev/null +++ b/tests/Foundatio.Aws.Tests/docker-compose.yml @@ -0,0 +1,10 @@ +services: + localstack: + # Pinned to the 3.x community edition: SQS/SNS run free, with no auth token (the rolling `latest`/Pro tag now + # requires a LocalStack license). + image: localstack/localstack:3 + ports: + - "4566:4566" + environment: + - SERVICES=sqs,sns + - DEBUG=0 diff --git a/tests/Foundatio.Redis.Tests/Foundatio.Redis.Tests.csproj b/tests/Foundatio.Redis.Tests/Foundatio.Redis.Tests.csproj new file mode 100644 index 000000000..3d2ab2138 --- /dev/null +++ b/tests/Foundatio.Redis.Tests/Foundatio.Redis.Tests.csproj @@ -0,0 +1,10 @@ + + + + $(TestingPlatformCommandLineArguments) --ignore-exit-code 8 + + + + + + diff --git a/tests/Foundatio.Redis.Tests/README.md b/tests/Foundatio.Redis.Tests/README.md new file mode 100644 index 000000000..cda720aab --- /dev/null +++ b/tests/Foundatio.Redis.Tests/README.md @@ -0,0 +1,25 @@ +# Foundatio.Redis.Tests + +Validates the temporary in-repo Redis providers against a real Redis: + +- **`RedisJobRuntimeStore`** — runs the shared `JobRuntimeStoreConformanceTests` suite (the same assertions the + in-memory reference store passes) plus `RedisJobStoreIntegrationTests` (delayed-send fallback + CRON end-to-end). +- **`RedisStreamsMessageTransport`** — runs the shared `MessageTransportConformanceTests` suite (pull, settlement, + visibility timeout, lock renewal, redelivery delay, dead-letter, provisioning, stats, topic fan-out; push/priority/ + expiration/delayed-delivery skip via capability gates) plus `RedisStreamsTransportIntegrationTests` (cross-instance + crash recovery, the core's retry/dead-letter machinery over Streams, and `PubSub` fan-out). + +## Running + +Start Redis and point the tests at it. Without the connection string every test is skipped. + +```sh +docker compose -f tests/Foundatio.Redis.Tests/docker-compose.yml up -d + +export FOUNDATIO_REDIS_CONNECTION_STRING=localhost:6399 +dotnet run --project tests/Foundatio.Redis.Tests +``` + +Each test runs under a unique key prefix, so concurrent runs and leftover keys never collide. The job-store suite +drives lease/expiry timing with a `FakeTimeProvider` (no real sleeps); the transport suite uses real, whole-second +timing windows (the same cross-transport windows the AWS suite uses). diff --git a/tests/Foundatio.Redis.Tests/RedisJobRuntimeStoreConformanceTests.cs b/tests/Foundatio.Redis.Tests/RedisJobRuntimeStoreConformanceTests.cs new file mode 100644 index 000000000..c541f8f5c --- /dev/null +++ b/tests/Foundatio.Redis.Tests/RedisJobRuntimeStoreConformanceTests.cs @@ -0,0 +1,22 @@ +using System; +using Foundatio.Jobs; +using Foundatio.Tests.Jobs; +using Xunit; + +namespace Foundatio.Redis.Tests; + +/// +/// Runs the shared conformance suite against a real Redis. Set +/// FOUNDATIO_REDIS_CONNECTION_STRING (e.g. localhost:6399 for the bundled docker-compose Redis) to run; +/// when it is not set every test is skipped. Each test gets a unique key prefix so runs never collide. Inheriting the +/// base [Fact]s means a new conformance check automatically runs against Redis with no override to forget. +/// +public class RedisJobRuntimeStoreConformanceTests : JobRuntimeStoreConformanceTests +{ + public RedisJobRuntimeStoreConformanceTests(ITestOutputHelper output) : base(output) { } + + protected override IJobRuntimeStore? CreateStore(TimeProvider timeProvider) => + RedisTestConnection.Multiplexer is { } connection + ? RedisTestConnection.CreateStore(connection, timeProvider) + : null; // not configured -> the base suite skips every test +} diff --git a/tests/Foundatio.Redis.Tests/RedisJobStoreIntegrationTests.cs b/tests/Foundatio.Redis.Tests/RedisJobStoreIntegrationTests.cs new file mode 100644 index 000000000..5d691efea --- /dev/null +++ b/tests/Foundatio.Redis.Tests/RedisJobStoreIntegrationTests.cs @@ -0,0 +1,283 @@ +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +namespace Foundatio.Redis.Tests; + +/// +/// End-to-end tests that wire the real messaging core / CRON scheduler on top of the Redis +/// and exercise the two paths the store exists to support but that the primitive-level conformance suite does not cover: +/// (1) a delayed send whose delay exceeds the transport's being +/// durably stored in Redis and drained by the dispatch pump when due, and (2) CRON occurrences being materialized, run, +/// retried/dead-lettered, and stale-reclaimed through Redis. +/// +/// Gated on FOUNDATIO_REDIS_CONNECTION_STRING; skips when unset. Each test isolates under a unique key prefix. +/// +public class RedisJobStoreIntegrationTests +{ + [Fact] + public async Task DelayedQueueSend_BeyondTransportLimit_StoresInRedisAndDrainsWhenDueAsync() + { + if (RedisTestConnection.Multiplexer is not { } connection) + { + Assert.Skip("FOUNDATIO_REDIS_CONNECTION_STRING not set."); + return; + } + + var cancellationToken = TestContext.Current.CancellationToken; + var now = DateTimeOffset.UtcNow; + + // Within the transport's advertised maximum: delivered natively, nothing is parked in Redis. + var nativeStore = RedisTestConnection.CreateStore(connection); + await using var nativeTransport = new CappedDelayTransport(TimeSpan.FromMinutes(15)); + await using var nativeQueue = new MessageBus(nativeTransport, new MessageBusOptions { RuntimeStore = nativeStore }); + var nativeProcessor = CreateProcessor(nativeStore, nativeTransport).Processor; + + await nativeQueue.SendAsync(new PreviewWorkItem { Data = "soon" }, new MessageSendOptions { Delay = TimeSpan.FromMinutes(5) }, cancellationToken); + Assert.Equal(1, nativeTransport.SendCount); + Assert.NotNull(nativeTransport.LastSendOptions?.DeliverAt); + Assert.Equal(0, await nativeProcessor.RunDueOccurrencesAsync(now.AddYears(1), cancellationToken: cancellationToken)); + + // Beyond the transport's maximum: routed into the Redis store rather than truncated to the broker ceiling. + var fallbackStore = RedisTestConnection.CreateStore(connection); + await using var fallbackTransport = new CappedDelayTransport(TimeSpan.FromMinutes(15)); + await using var fallbackQueue = new MessageBus(fallbackTransport, new MessageBusOptions { RuntimeStore = fallbackStore }); + var fallbackProcessor = CreateProcessor(fallbackStore, fallbackTransport).Processor; + + await fallbackQueue.SendAsync(new PreviewWorkItem { Data = "later" }, new MessageSendOptions { Delay = TimeSpan.FromHours(1) }, cancellationToken); + Assert.Equal(0, fallbackTransport.SendCount); + + // Durably parked in Redis and time-gated: a drain before the due time claims nothing; only when due does the + // pump pull it from Redis and hand it to the transport. + Assert.Equal(0, await fallbackProcessor.RunDueOccurrencesAsync(now, cancellationToken: cancellationToken)); + Assert.Equal(0, fallbackTransport.SendCount); + + Assert.Equal(1, await fallbackProcessor.RunDueOccurrencesAsync(now.AddHours(2), cancellationToken: cancellationToken)); + Assert.Equal(1, fallbackTransport.SendCount); + + var deliveredContext = new TaskCompletionSource>(TaskCreationOptions.RunContinuationsAsynchronously); + await using var subscription = await fallbackQueue.SubscribeAsync((context, _) => + { + deliveredContext.TrySetResult(context); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { AckMode = AckMode.Manual }, cancellationToken); + + var delivered = await deliveredContext.Task.WaitAsync(TimeSpan.FromSeconds(10), cancellationToken); + Assert.Equal("later", delivered.Message.Data); + await delivered.CompleteAsync(cancellationToken); + } + + [Fact] + public async Task CronOccurrence_MaterializesRunsAndDedupesThroughRedisAsync() + { + if (RedisTestConnection.Multiplexer is not { } connection) + { + Assert.Skip("FOUNDATIO_REDIS_CONNECTION_STRING not set."); + return; + } + + var cancellationToken = TestContext.Current.CancellationToken; + var store = RedisTestConnection.CreateStore(connection); + var scheduler = new InMemoryJobScheduler(); + var (processor, probe) = CreateProcessor(store, scheduler); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "nightly", + Cron = "* * * * *", + JobType = typeof(ProbeJob) + }, cancellationToken); + + // Materialize: one occurrence is written to Redis as a Scheduled JobState + a JobOccurrence dispatch. + var first = await processor.EnqueueDueOccurrencesAsync(now, cancellationToken); + var dispatch = Assert.Single(first); + Assert.Equal("nightly:20260101000000:global", dispatch.DispatchId); + Assert.Equal(ScheduledDispatchKind.JobOccurrence, dispatch.Kind); + Assert.Equal("nightly", dispatch.Headers["job.name"]); + + var scheduled = await store.GetAsync(dispatch.JobId!, cancellationToken); + Assert.NotNull(scheduled); + Assert.Equal(JobStatus.Scheduled, scheduled.Status); + Assert.Equal(new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero), scheduled.ScheduledForUtc); + + // Deterministic occurrence id dedupes against the Redis row: a second materialize pass at the same time is a no-op. + Assert.Empty(await processor.EnqueueDueOccurrencesAsync(now, cancellationToken)); + + // Claim from Redis and run: the occurrence completes and the run is recorded once. + Assert.Equal(1, await processor.RunDueOccurrencesAsync(now, cancellationToken: cancellationToken)); + Assert.Equal(1, probe.RunCount); + + var completed = await store.GetAsync(dispatch.JobId!, cancellationToken); + Assert.NotNull(completed); + Assert.Equal(JobStatus.Completed, completed.Status); + Assert.Equal(1, completed.Attempt); + Assert.Equal(100, completed.Progress); + + // The dispatch was completed (removed) in Redis, so a later drain finds nothing. + Assert.Equal(0, await processor.RunDueOccurrencesAsync(now.AddMinutes(1), cancellationToken: cancellationToken)); + } + + [Fact] + public async Task CronOccurrence_RetryDeadLetterAndStaleReclaimThroughRedisAsync() + { + if (RedisTestConnection.Multiplexer is not { } connection) + { + Assert.Skip("FOUNDATIO_REDIS_CONNECTION_STRING not set."); + return; + } + + var cancellationToken = TestContext.Current.CancellationToken; + var store = RedisTestConnection.CreateStore(connection); + var scheduler = new InMemoryJobScheduler(); + var (processor, probe) = CreateProcessor(store, scheduler); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + + // (a) Retry-then-dead-letter: a failing occurrence is rescheduled in Redis until its retry budget is spent. + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "flaky", + Cron = "* * * * *", + JobType = typeof(FailingJob), + MaxRetries = 1 + }, cancellationToken); + var flaky = Assert.Single(await processor.EnqueueDueOccurrencesAsync(now, cancellationToken)); + + Assert.Equal(0, await processor.RunDueOccurrencesAsync(now, cancellationToken: cancellationToken)); + var retried = await store.GetAsync(flaky.JobId!, cancellationToken); + Assert.NotNull(retried); + Assert.Equal(JobStatus.Scheduled, retried.Status); + Assert.Equal(1, retried.Attempt); + + Assert.Equal(1, await processor.RunDueOccurrencesAsync(now.AddMinutes(2), cancellationToken: cancellationToken)); + var deadlettered = await store.GetAsync(flaky.JobId!, cancellationToken); + Assert.NotNull(deadlettered); + Assert.Equal(JobStatus.DeadLettered, deadlettered.Status); + Assert.Equal(2, deadlettered.Attempt); + + // (b) Stale reclaim: an occurrence stuck in Processing under a dead node with an expired lease is reclaimed + // (via the Redis CAS reclaim) and run to completion by the live node. + const string jobId = "nightly:20260101000000:global"; + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "nightly", + Cron = "* * * * *", + JobType = typeof(ProbeJob), + MaxRetries = 1 + }, cancellationToken); + await store.CreateIfAbsentAsync(new JobState + { + JobId = jobId, + Name = "nightly", + Status = JobStatus.Processing, + Attempt = 1, + NodeId = "node-b", + LeaseExpiresUtc = now.AddMinutes(-1), + ScheduledForUtc = now.AddSeconds(-30) + }, cancellationToken); + await store.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = jobId, + Kind = ScheduledDispatchKind.JobOccurrence, + JobName = "nightly", + Body = Array.Empty(), + DueUtc = now, + JobId = jobId + }, cancellationToken); + + Assert.Equal(1, await processor.RunDueOccurrencesAsync(now, cancellationToken: cancellationToken)); + var reclaimed = await store.GetAsync(jobId, cancellationToken); + Assert.NotNull(reclaimed); + Assert.Equal(JobStatus.Completed, reclaimed.Status); + Assert.Equal(2, reclaimed.Attempt); + Assert.Equal(1, probe.RunCount); + } + + private static (JobScheduleProcessor Processor, Probe Probe) CreateProcessor(IJobRuntimeStore store, IMessageTransport? transport = null) + => CreateProcessor(store, new InMemoryJobScheduler(), transport); + + private static (JobScheduleProcessor Processor, Probe Probe) CreateProcessor(IJobRuntimeStore store, IJobScheduler scheduler, IMessageTransport? transport = null) + { + var probe = new Probe(); + var serviceProvider = new ServiceCollection().AddSingleton(probe).BuildServiceProvider(); + var worker = new JobWorker(store, serviceProvider, nodeId: "node-a"); + return (new JobScheduleProcessor(scheduler, store, worker, nodeId: "node-a", transport: transport), probe); + } + + private sealed class Probe + { + private int _runCount; + public int RunCount => Volatile.Read(ref _runCount); + public void Record() => Interlocked.Increment(ref _runCount); + } + + private sealed class ProbeJob(Probe probe) : IJob + { + public Task RunAsync(JobExecutionContext context) + { + context.CancellationToken.ThrowIfCancellationRequested(); + probe.Record(); + return Task.FromResult(JobResult.Success); + } + } + + private sealed class FailingJob : IJob + { + public Task RunAsync(JobExecutionContext context) + { + context.CancellationToken.ThrowIfCancellationRequested(); + return Task.FromResult(JobResult.FromException(new InvalidOperationException("boom"))); + } + } + + private sealed class PreviewWorkItem + { + public string? Data { get; set; } + } + + // Minimal pull transport with a configurable native delayed-delivery ceiling, so a delay beyond the cap is forced + // through the runtime store (mirrors the fixture used by the in-memory MessageBus tests). + private sealed class CappedDelayTransport : IMessageTransport, ISupportsPull, ITransportInfo + { + private readonly Queue _entries = new(); + + public CappedDelayTransport(TimeSpan? maxDeliveryDelay) => MaxDeliveryDelay = maxDeliveryDelay; + + public TimeSpan? MaxDeliveryDelay { get; } + public int SendCount { get; private set; } + public TransportSendOptions? LastSendOptions { get; private set; } + + public DeliveryGuarantee DeliveryGuarantee => DeliveryGuarantee.AtLeastOnce; + public IReadOnlySet SupportedRoles => new HashSet { DestinationRole.Queue }; + + public TransportCapabilities GetCapabilities(DestinationRole role) => + new() { DelayedDelivery = true, MaxDeliveryDelay = MaxDeliveryDelay }; + + public Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + SendCount += messages.Count; + LastSendOptions = options; + var items = new SendItemResult[messages.Count]; + for (int i = 0; i < messages.Count; i++) + { + string id = messages[i].MessageId ?? Guid.NewGuid().ToString("N"); + _entries.Enqueue(new TransportEntry { Id = id, Destination = destination, Body = messages[i].Body, Headers = messages[i].Headers, Receipt = new Receipt() }); + items[i] = new SendItemResult { MessageId = id }; + } + + return Task.FromResult(new SendResult { Items = items }); + } + + public Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct) + => Task.FromResult>(_entries.Count > 0 ? [_entries.Dequeue()] : []); + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + public ValueTask DisposeAsync() => ValueTask.CompletedTask; + } +} diff --git a/tests/Foundatio.Redis.Tests/RedisStreamsTransportConformanceTests.cs b/tests/Foundatio.Redis.Tests/RedisStreamsTransportConformanceTests.cs new file mode 100644 index 000000000..c935c5b8b --- /dev/null +++ b/tests/Foundatio.Redis.Tests/RedisStreamsTransportConformanceTests.cs @@ -0,0 +1,30 @@ +using System; +using Foundatio.Messaging; +using Foundatio.Tests.Messaging; +using Xunit; + +namespace Foundatio.Redis.Tests; + +/// +/// Runs the shared transport conformance suite against the Redis Streams transport. Set +/// FOUNDATIO_REDIS_CONNECTION_STRING (e.g. localhost:6399) to run; skips when unset. A unique key prefix +/// per transport isolates each test's streams. Capabilities Streams does not provide (push delivery, per-message +/// priority, per-message expiration, native scheduled delivery) are skipped by the base suite's capability gates. +/// +public class RedisStreamsTransportConformanceTests : MessageTransportConformanceTests +{ + public RedisStreamsTransportConformanceTests(ITestOutputHelper output) : base(output) { } + + protected override IMessageTransport? CreateTransport() + { + if (RedisTestConnection.Multiplexer is not { } connection) + return null; // not configured -> the base suite skips every test + + return new RedisStreamsMessageTransport(new RedisStreamsMessageTransportOptions + { + ConnectionMultiplexer = connection, + KeyPrefix = $"fnd-conf:{Guid.NewGuid():N}:" + }); + } + +} diff --git a/tests/Foundatio.Redis.Tests/RedisStreamsTransportIntegrationTests.cs b/tests/Foundatio.Redis.Tests/RedisStreamsTransportIntegrationTests.cs new file mode 100644 index 000000000..e35bfab4c --- /dev/null +++ b/tests/Foundatio.Redis.Tests/RedisStreamsTransportIntegrationTests.cs @@ -0,0 +1,260 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Xunit; + +namespace Foundatio.Redis.Tests; + +/// +/// End-to-end tests for the Redis Streams transport that the cross-transport conformance suite can't express: at-least-once +/// recovery across two consumer instances, the core's retry/dead-letter machinery driving the transport, and topic +/// fan-out through the facade. Gated on FOUNDATIO_REDIS_CONNECTION_STRING; unique key prefix +/// per test. +/// +public class RedisStreamsTransportIntegrationTests +{ + private static RedisStreamsMessageTransport CreateTransport(StackExchange.Redis.IConnectionMultiplexer connection, string prefix, string? consumer = null) => + new(new RedisStreamsMessageTransportOptions + { + ConnectionMultiplexer = connection, + KeyPrefix = prefix, + ConsumerName = consumer + }); + + private static string NewPrefix() => $"fnd-it:{Guid.NewGuid():N}:"; + + [Fact] + public async Task CrashedConsumer_LeaseLapses_AnotherInstanceReclaimsAndCompletesAsync() + { + if (RedisTestConnection.Multiplexer is not { } connection) + { + Assert.Skip("FOUNDATIO_REDIS_CONNECTION_STRING not set."); + return; + } + + var ct = TestContext.Current.CancellationToken; + string prefix = NewPrefix(); + var visibility = TimeSpan.FromSeconds(2); + + // Both instances share the same key prefix so they operate on the same streams (the lease lives in Redis). + await using var nodeA = CreateTransport(connection, prefix, "node-a"); + await using var nodeB = CreateTransport(connection, prefix, "node-b"); + + await nodeA.EnsureAsync([new DestinationDeclaration { Address = DestinationAddress.ForQueue("work") }], ct); + await nodeA.SendAsync(DestinationAddress.ForQueue("work"), [Message("survive-me")], new TransportSendOptions(), ct); + + // node-a receives and then "crashes" — it never settles the message. + var heldByA = Assert.Single(await nodeA.ReceiveAsync(DestinationAddress.ForQueue("work"), new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, visibility, ct)); + Assert.Equal(1, heldByA.DeliveryCount); + + // While node-a's lease is live, node-b must not see it. + Assert.Empty(await nodeB.ReceiveAsync(DestinationAddress.ForQueue("work"), new ReceiveRequest { MaxWaitTime = TimeSpan.FromMilliseconds(100) }, visibility, ct)); + + // After the lease lapses, node-b reclaims the in-flight message (lease state lives in Redis) and completes it. + var reclaimedByB = Assert.Single(await nodeB.ReceiveAsync(DestinationAddress.ForQueue("work"), new ReceiveRequest { MaxWaitTime = visibility + TimeSpan.FromSeconds(5) }, visibility, ct)); + Assert.Equal(heldByA.Id, reclaimedByB.Id); + Assert.Equal(2, reclaimedByB.DeliveryCount); + Assert.Equal("survive-me", System.Text.Encoding.UTF8.GetString(reclaimedByB.Body.Span)); + await nodeB.CompleteAsync(reclaimedByB, ct); + + var stats = await nodeB.GetStatsAsync(DestinationAddress.ForQueue("work"), ct); + Assert.Equal(0, stats.Queued); + Assert.Equal(0, stats.Working); + } + + [Fact] + public async Task Core_RetriesFailedHandler_ThenDeadLettersAfterMaxAttemptsAsync() + { + if (RedisTestConnection.Multiplexer is not { } connection) + { + Assert.Skip("FOUNDATIO_REDIS_CONNECTION_STRING not set."); + return; + } + + var ct = TestContext.Current.CancellationToken; + var transport = CreateTransport(connection, NewPrefix()); + await using var queue = new MessageBus(transport, new MessageBusOptions()); + + // (a) A handler that throws once is redelivered (via the transport) and succeeds on the second attempt — the + // core's retry machinery works unchanged over Streams. + int retryAttempts = 0; + var succeeded = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + await using var retryConsumer = await queue.SubscribeAsync((message, _) => + { + int attempt = Interlocked.Increment(ref retryAttempts); + if (attempt == 1) + throw new InvalidOperationException("first attempt fails"); + + Assert.Equal(2, message.Attempts); + succeeded.TrySetResult(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { MaxAttempts = 3, RedeliveryBackoff = _ => TimeSpan.FromMilliseconds(200) }, ct); + + // (b) A handler that always throws is dead-lettered once its attempt budget is spent. + await using var poisonConsumer = await queue.SubscribeAsync((_, _) => + throw new InvalidOperationException("always fails"), + new MessageSubscriptionOptions { MaxAttempts = 2, RedeliveryBackoff = _ => TimeSpan.FromMilliseconds(100) }, ct); + + await queue.SendAsync(new RetryItem { Data = "retry" }, cancellationToken: ct); + await queue.SendAsync(new PoisonItem { Data = "poison" }, cancellationToken: ct); + + await succeeded.Task.WaitAsync(TimeSpan.FromSeconds(30), ct); + Assert.Equal(2, Volatile.Read(ref retryAttempts)); + + // The poison message lands in the dead-letter stream after exhausting its 2 attempts. + MessageDestinationStats stats = await transport.GetStatsAsync(DestinationAddress.ForQueue("streams-poison"), ct); + for (int i = 0; i < 100 && stats.Deadletter == 0; i++) + { + await Task.Delay(100, ct); + stats = await transport.GetStatsAsync(DestinationAddress.ForQueue("streams-poison"), ct); + } + + Assert.Equal(1, stats.Deadletter); + Assert.Equal(0, stats.Working); + + // The poison payload is inspectable in the dead-letter stream with a reason recorded by the core. + var deadLettered = Assert.Single(await transport.ReceiveDeadLetteredAsync(DestinationAddress.ForQueue("streams-poison"), new ReceiveRequest { MaxMessages = 10 }, ct)); + Assert.NotEmpty(deadLettered.Headers[KnownHeaders.DeadLetterReason]); + } + + [Fact] + public async Task PubSub_PublishToTopic_FansOutToEverySubscriptionAsync() + { + if (RedisTestConnection.Multiplexer is not { } connection) + { + Assert.Skip("FOUNDATIO_REDIS_CONNECTION_STRING not set."); + return; + } + + var ct = TestContext.Current.CancellationToken; + var transport = CreateTransport(connection, NewPrefix()); + await using var pubsub = new MessageBus(transport, new MessageBusOptions()); + + var receivedByA = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var receivedByB = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + + await using var subA = await pubsub.SubscribeAsync((message, _) => + { + receivedByA.TrySetResult(message.Message.Data ?? ""); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "sub-a" }, ct); + + await using var subB = await pubsub.SubscribeAsync((message, _) => + { + receivedByB.TrySetResult(message.Message.Data ?? ""); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "sub-b" }, ct); + + await pubsub.PublishAsync(new FanItem { Data = "broadcast" }, cancellationToken: ct); + + // Each named subscription is its own consumer group, so both receive an independent copy. Delivery is + // poll-driven across two subscriptions (the core pull-fallback loop), so allow generous headroom for the whole + // conformance suite hammering the same Redis concurrently. + await Task.WhenAll(receivedByA.Task, receivedByB.Task).WaitAsync(TimeSpan.FromSeconds(30), ct); + Assert.Equal("broadcast", await receivedByA.Task); + Assert.Equal("broadcast", await receivedByB.Task); + } + + [Fact] + public async Task MessageBus_SendAndPublishSameType_StayIsolatedAsync() + { + if (RedisTestConnection.Multiplexer is not { } connection) + { + Assert.Skip("FOUNDATIO_REDIS_CONNECTION_STRING not set."); + return; + } + + var ct = TestContext.Current.CancellationToken; + var transport = CreateTransport(connection, NewPrefix()); + await using var bus = new MessageBus(transport, new MessageBusOptions()); + + // One subscription listens on both of the type's channels. Send targets the queue-role stream and Publish the + // topic-role stream, so the same route name must never cross-deliver: exactly one delivery per verb. (A shared + // stream would deliver each message through BOTH channels — 4 deliveries instead of 2.) + var sent = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + var published = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + int deliveries = 0; + + await using var subscription = await bus.SubscribeAsync((message, _) => + { + Interlocked.Increment(ref deliveries); + if (message.Message.Data == "for-one") + sent.TrySetResult(message.Message.Data); + else + published.TrySetResult(message.Message.Data ?? ""); + return Task.CompletedTask; + }, cancellationToken: ct); + + await bus.SendAsync(new DualItem { Data = "for-one" }, cancellationToken: ct); + await bus.PublishAsync(new DualItem { Data = "for-all" }, cancellationToken: ct); + + await Task.WhenAll(sent.Task, published.Task).WaitAsync(TimeSpan.FromSeconds(30), ct); + Assert.Equal("for-one", await sent.Task); + Assert.Equal("for-all", await published.Task); + + // Give any cross-delivery a moment to surface, then assert exactly one delivery per verb. + await Task.Delay(500, ct); + Assert.Equal(2, Volatile.Read(ref deliveries)); + } + + [Fact] + public async Task Publish_CompletedByOneGroup_StillDeliveredToSlowerGroupAsync() + { + if (RedisTestConnection.Multiplexer is not { } connection) + { + Assert.Skip("FOUNDATIO_REDIS_CONNECTION_STRING not set."); + return; + } + + var ct = TestCancellation(); + await using var transport = CreateTransport(connection, NewPrefix()); + + await transport.EnsureAsync( + [ + new DestinationDeclaration { Address = DestinationAddress.ForTopic("iso-topic") }, + new DestinationDeclaration { Address = DestinationAddress.ForSubscription("iso-topic", "sub-a") }, + new DestinationDeclaration { Address = DestinationAddress.ForSubscription("iso-topic", "sub-b") } + ], ct); + + await transport.SendAsync(DestinationAddress.ForTopic("iso-topic"), [Message("retained")], new TransportSendOptions(), ct); + + // Group A reads and completes FIRST; the entry must remain on the topic stream for group B (completing must + // not delete a shared topic entry other groups haven't read yet). + var byA = Assert.Single(await transport.ReceiveAsync(DestinationAddress.ForSubscription("iso-topic", "sub-a"), new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, ct)); + await transport.CompleteAsync(byA, ct); + + var byB = Assert.Single(await transport.ReceiveAsync(DestinationAddress.ForSubscription("iso-topic", "sub-b"), new ReceiveRequest { MaxWaitTime = TimeSpan.FromSeconds(2) }, ct)); + Assert.Equal("retained", System.Text.Encoding.UTF8.GetString(byB.Body.Span)); + await transport.CompleteAsync(byB, ct); + } + + private static CancellationToken TestCancellation() => TestContext.Current.CancellationToken; + + private static TransportMessage Message(string body) => + new() { Body = System.Text.Encoding.UTF8.GetBytes(body) }; + + [MessageRoute("streams-dual")] + private sealed class DualItem + { + public string? Data { get; set; } + } + + [MessageRoute("streams-retry")] + private sealed class RetryItem + { + public string? Data { get; set; } + } + + [MessageRoute("streams-poison")] + private sealed class PoisonItem + { + public string? Data { get; set; } + } + + [MessageRoute("streams-topic")] + private sealed class FanItem + { + public string? Data { get; set; } + } +} diff --git a/tests/Foundatio.Redis.Tests/RedisTestConnection.cs b/tests/Foundatio.Redis.Tests/RedisTestConnection.cs new file mode 100644 index 000000000..41a06dfc5 --- /dev/null +++ b/tests/Foundatio.Redis.Tests/RedisTestConnection.cs @@ -0,0 +1,30 @@ +using System; +using Foundatio.Jobs; +using StackExchange.Redis; + +namespace Foundatio.Redis.Tests; + +/// +/// Shared, lazily-opened Redis connection for the Redis test suites. Gated on +/// FOUNDATIO_REDIS_CONNECTION_STRING (e.g. localhost:6399 for the bundled docker-compose Redis); when it +/// is unset is null and the suites skip every test. +/// +internal static class RedisTestConnection +{ + private static readonly Lazy Shared = new(() => + { + string? connectionString = Environment.GetEnvironmentVariable("FOUNDATIO_REDIS_CONNECTION_STRING"); + return String.IsNullOrEmpty(connectionString) ? null : ConnectionMultiplexer.Connect(connectionString); + }); + + public static IConnectionMultiplexer? Multiplexer => Shared.Value; + + /// Creates a store under a unique key prefix so concurrent tests and leftover keys never collide. + public static RedisJobRuntimeStore CreateStore(IConnectionMultiplexer connection, TimeProvider? timeProvider = null) => + new(new RedisJobRuntimeStoreOptions + { + ConnectionMultiplexer = connection, + KeyPrefix = $"fnd-it:{Guid.NewGuid():N}:", + TimeProvider = timeProvider + }); +} diff --git a/tests/Foundatio.Redis.Tests/docker-compose.yml b/tests/Foundatio.Redis.Tests/docker-compose.yml new file mode 100644 index 000000000..d41de3cdd --- /dev/null +++ b/tests/Foundatio.Redis.Tests/docker-compose.yml @@ -0,0 +1,6 @@ +services: + redis: + image: redis:7-alpine + # Mapped to 6399 to avoid clashing with a local Redis on the default 6379. + ports: + - "6399:6379" diff --git a/tests/Foundatio.Tests/Caching/InMemoryHybridCacheClientTests.cs b/tests/Foundatio.Tests/Caching/InMemoryHybridCacheClientTests.cs index bf29b93c0..53e0a4b5c 100644 --- a/tests/Foundatio.Tests/Caching/InMemoryHybridCacheClientTests.cs +++ b/tests/Foundatio.Tests/Caching/InMemoryHybridCacheClientTests.cs @@ -1,6 +1,6 @@ using System.Threading.Tasks; using Foundatio.Caching; -using Foundatio.Messaging; +using Foundatio.Messaging.Legacy; using Foundatio.Utility; using Microsoft.Extensions.Logging; using Xunit; diff --git a/tests/Foundatio.Tests/DeclarativeRegistrationTests.cs b/tests/Foundatio.Tests/DeclarativeRegistrationTests.cs new file mode 100644 index 000000000..ab20b430e --- /dev/null +++ b/tests/Foundatio.Tests/DeclarativeRegistrationTests.cs @@ -0,0 +1,289 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Xunit; + +namespace Foundatio.Tests; + +public class DeclarativeRegistrationTests +{ + [Fact] + public async Task AddHandler_SendGoesToOneHandlerAndPublishReachesSubscriptionAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var probe = new HandlerProbe(); + + var services = new ServiceCollection(); + services.AddLogging(); + services.AddSingleton(probe); + services.AddFoundatio() + .Messaging.UseInMemory() + .Messaging.AddHandler() // class handler + .Messaging.AddHandler((context, _) => { probe.Record($"task:{context.Message.Id}"); return Task.CompletedTask; }); // delegate handler + + await using var provider = services.BuildServiceProvider(); + var hosted = provider.GetServices().ToList(); + Assert.Single(hosted); // one auto-registered hosted service drives every handler + + foreach (var service in hosted) + await service.StartAsync(cancellationToken); + + try + { + var bus = provider.GetRequiredService(); + + // The caller's verb decides delivery; the same registration serves both. + await bus.SendAsync(new HandledOrder { Id = "sent" }, cancellationToken: cancellationToken); + await bus.PublishAsync(new HandledOrder { Id = "published" }, cancellationToken: cancellationToken); + await bus.SendAsync(new HandledTask { Id = "t1" }, cancellationToken: cancellationToken); + + Assert.True(await probe.WaitForAsync(3, TimeSpan.FromSeconds(10)), $"handled: {String.Join(",", probe.Events)}"); + Assert.Contains("order:sent", probe.Events); + Assert.Contains("order:published", probe.Events); + Assert.Contains("task:t1", probe.Events); + + // Same type sent AND published: exactly one delivery per verb — the queue and topic namespaces are + // segregated, so a send is never fanned out and a publish is never consumed as queue work. + Assert.Equal(3, probe.Events.Count); + } + finally + { + foreach (var service in hosted) + await service.StopAsync(cancellationToken); + } + } + + [Fact] + public async Task AddHandler_PublishIsOncePerServiceUnlessPerInstanceAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + + // Two service providers sharing one transport simulate two scaled instances of the same service. + var sharedProbe = new HandlerProbe(); + var instanceA = BuildInstance(transport, sharedProbe); + var instanceB = BuildInstance(transport, sharedProbe); + + await using (instanceA.Provider) + await using (instanceB.Provider) + { + await StartAsync(instanceA, cancellationToken); + await StartAsync(instanceB, cancellationToken); + + try + { + var bus = instanceA.Provider.GetRequiredService(); + + // Default subscription = service identity, shared by both instances => they compete: one copy total. + await bus.PublishAsync(new HandledEvent { Id = "shared" }, cancellationToken: cancellationToken); + Assert.True(await sharedProbe.WaitForAsync(1, TimeSpan.FromSeconds(10)), $"handled: {String.Join(",", sharedProbe.Events)}"); + await Task.Delay(250, cancellationToken); + Assert.Single(sharedProbe.Events, e => e.StartsWith("event:", StringComparison.Ordinal)); + + // PerInstance handlers each take a unique subscription => every instance receives its own copy. + await bus.PublishAsync(new HandledBroadcast { Id = "all" }, cancellationToken: cancellationToken); + Assert.True(await sharedProbe.WaitForAsync(3, TimeSpan.FromSeconds(10)), $"handled: {String.Join(",", sharedProbe.Events)}"); + Assert.Equal(2, sharedProbe.Events.Count(e => e.StartsWith("broadcast:", StringComparison.Ordinal))); + } + finally + { + await StopAsync(instanceA, cancellationToken); + await StopAsync(instanceB, cancellationToken); + } + } + } + + [Fact] + public async Task AddHandler_TwoHandlerClassesForOneType_EachGetsPublishedAndSendReachesOneAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var probe = new HandlerProbe(); + + var services = new ServiceCollection(); + services.AddLogging(); + services.AddSingleton(probe); + services.AddFoundatio() + .Messaging.UseInMemory() + .Messaging.AddHandler() + .Messaging.AddHandler(); + + await using var provider = services.BuildServiceProvider(); + var hosted = provider.GetServices().ToList(); + foreach (var service in hosted) + await service.StartAsync(cancellationToken); + + try + { + var bus = provider.GetRequiredService(); + + // An event reaches EVERY handler class (each is its own subscriber group). + await bus.PublishAsync(new HandledEvent { Id = "e1" }, cancellationToken: cancellationToken); + Assert.True(await probe.WaitForAsync(2, TimeSpan.FromSeconds(10)), $"handled: {String.Join(",", probe.Events)}"); + Assert.Contains("event:e1", probe.Events); + Assert.Contains("second:e1", probe.Events); + + // A command reaches exactly ONE handler (competing consumers on the type's send channel). + await bus.SendAsync(new HandledEvent { Id = "s1" }, cancellationToken: cancellationToken); + Assert.True(await probe.WaitForAsync(3, TimeSpan.FromSeconds(10)), $"handled: {String.Join(",", probe.Events)}"); + await Task.Delay(250, cancellationToken); + Assert.Equal(1, probe.Events.Count(e => e.EndsWith(":s1", StringComparison.Ordinal))); + } + finally + { + foreach (var service in hosted) + await service.StopAsync(cancellationToken); + } + } + + [Fact] + public async Task AddCronJob_RegistersDefinitionAndSchedulesWhenPumpStartsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + + var services = new ServiceCollection(); + services.AddLogging(); + services.AddFoundatio() + .Jobs.UseInMemoryRuntime() + .Jobs.AddCronJob("* * * * *", o => o.Scope = ScheduledJobScope.PerNode); + + await using var provider = services.BuildServiceProvider(); + + // The builder records the schedule as a DI singleton with the requested scope and a type-derived name. + var definition = Assert.Single(provider.GetServices()); + Assert.Equal(typeof(CronProbeJob), definition.JobType); + Assert.Equal(ScheduledJobScope.PerNode, definition.Scope); + Assert.Equal(nameof(CronProbeJob), definition.Name); + + // Starting the runtime pump schedules registered CRON jobs into the scheduler — no manual ScheduleAsync call. + var hosted = provider.GetServices().ToList(); + foreach (var service in hosted) + await service.StartAsync(cancellationToken); + + try + { + var scheduler = provider.GetRequiredService(); + ScheduledJobDefinition? scheduled = null; + long deadline = Environment.TickCount64 + 10_000; + while (Environment.TickCount64 < deadline) + { + scheduled = (await scheduler.GetSchedulesAsync(cancellationToken)).FirstOrDefault(s => s.Name == nameof(CronProbeJob)); + if (scheduled is not null) + break; + await Task.Delay(25, cancellationToken); + } + + Assert.NotNull(scheduled); + Assert.Equal(ScheduledJobScope.PerNode, scheduled!.Scope); + } + finally + { + foreach (var service in hosted) + await service.StopAsync(cancellationToken); + } + } + + private static (ServiceProvider Provider, List Hosted) BuildInstance(InMemoryMessageTransport transport, HandlerProbe probe) + { + var services = new ServiceCollection(); + services.AddLogging(); + services.AddSingleton(probe); + services.AddFoundatio() + .Messaging.UseTransport(transport) + .Messaging.AddHandler() + .Messaging.AddHandler(o => o.PerInstance = true); + + var provider = services.BuildServiceProvider(); + return (provider, provider.GetServices().ToList()); + } + + private static async Task StartAsync((ServiceProvider Provider, List Hosted) instance, CancellationToken cancellationToken) + { + foreach (var service in instance.Hosted) + await service.StartAsync(cancellationToken); + } + + private static async Task StopAsync((ServiceProvider Provider, List Hosted) instance, CancellationToken cancellationToken) + { + foreach (var service in instance.Hosted) + await service.StopAsync(cancellationToken); + } + + private sealed class HandlerProbe + { + private readonly ConcurrentBag _events = new(); + public IReadOnlyCollection Events => _events; + public void Record(string value) => _events.Add(value); + + public async Task WaitForAsync(int count, TimeSpan timeout) + { + long deadline = Environment.TickCount64 + (long)timeout.TotalMilliseconds; + while (Environment.TickCount64 < deadline) + { + if (_events.Count >= count) + return true; + await Task.Delay(25); + } + return _events.Count >= count; + } + } + + [MessageRoute("declarative-orders")] + public class HandledOrder { public string Id { get; set; } = ""; } + + [MessageRoute("declarative-tasks")] + public class HandledTask { public string Id { get; set; } = ""; } + + [MessageRoute("declarative-events")] + public class HandledEvent { public string Id { get; set; } = ""; } + + [MessageRoute("declarative-broadcasts")] + public class HandledBroadcast { public string Id { get; set; } = ""; } + + private sealed class OrderHandler(HandlerProbe probe) : IMessageHandler + { + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + { + probe.Record($"order:{context.Message.Id}"); + return Task.CompletedTask; + } + } + + private sealed class EventHandler(HandlerProbe probe) : IMessageHandler + { + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + { + probe.Record($"event:{context.Message.Id}"); + return Task.CompletedTask; + } + } + + private sealed class SecondEventHandler(HandlerProbe probe) : IMessageHandler + { + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + { + probe.Record($"second:{context.Message.Id}"); + return Task.CompletedTask; + } + } + + private sealed class BroadcastHandler(HandlerProbe probe) : IMessageHandler + { + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + { + probe.Record($"broadcast:{context.Message.Id}"); + return Task.CompletedTask; + } + } + + private sealed class CronProbeJob : IJob + { + public Task RunAsync(JobExecutionContext context) => Task.FromResult(JobResult.Success); + } +} diff --git a/tests/Foundatio.Tests/Foundatio.Tests.csproj b/tests/Foundatio.Tests/Foundatio.Tests.csproj index cd934dc96..fb17d2e29 100644 --- a/tests/Foundatio.Tests/Foundatio.Tests.csproj +++ b/tests/Foundatio.Tests/Foundatio.Tests.csproj @@ -9,6 +9,7 @@ + diff --git a/tests/Foundatio.Tests/Jobs/InMemoryJobRuntimeStoreTests.cs b/tests/Foundatio.Tests/Jobs/InMemoryJobRuntimeStoreTests.cs new file mode 100644 index 000000000..6d6c1a212 --- /dev/null +++ b/tests/Foundatio.Tests/Jobs/InMemoryJobRuntimeStoreTests.cs @@ -0,0 +1,14 @@ +using System; +using Foundatio.Jobs; +using Xunit; + +namespace Foundatio.Tests.Jobs; + +// Inherits every [Fact] from JobRuntimeStoreConformanceTests, so a new conformance check automatically runs here with +// no per-test override to forget. +public class InMemoryJobRuntimeStoreTests : JobRuntimeStoreConformanceTests +{ + public InMemoryJobRuntimeStoreTests(ITestOutputHelper output) : base(output) { } + + protected override IJobRuntimeStore CreateStore(TimeProvider timeProvider) => new InMemoryJobRuntimeStore(timeProvider); +} diff --git a/tests/Foundatio.Tests/Jobs/JobRuntimeTests.cs b/tests/Foundatio.Tests/Jobs/JobRuntimeTests.cs new file mode 100644 index 000000000..4d5119a90 --- /dev/null +++ b/tests/Foundatio.Tests/Jobs/JobRuntimeTests.cs @@ -0,0 +1,549 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +namespace Foundatio.Tests.Jobs; + +public class JobRuntimeTests +{ + [Fact] + public async Task RunAsync_WithExecutionContext_ReportsProgressAndIdentityAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + await using var serviceProvider = new ServiceCollection().BuildServiceProvider(); + var worker = new JobWorker(store, serviceProvider, nodeId: "ctx-node"); + + await store.CreateIfAbsentAsync(new JobState + { + JobId = "ctx-job", + Name = "ctx", + JobType = typeof(ProgressJob).FullName, + Status = JobStatus.Queued + }, cancellationToken); + + Assert.True(await worker.RunAsync("ctx-job", cancellationToken)); + + var state = await store.GetAsync("ctx-job", cancellationToken); + Assert.Equal(JobStatus.Completed, state!.Status); + Assert.Equal(100, state.Progress); // a completed job is 100%; the worker sets this on success + // The job wrote its context identity + attempt into the progress message (preserved through completion), + // proving the store-backed context is wired through to job code. + Assert.Equal("ctx-job:1", state.ProgressMessage); + } + + [Fact] + public async Task RecoverStaleAsync_ReclaimsExpiredProcessingJobsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + await using var serviceProvider = new ServiceCollection().BuildServiceProvider(); + var worker = new JobWorker(store, serviceProvider, nodeId: "recovery-node"); + + var expired = DateTimeOffset.UtcNow.AddMinutes(-5); + + // A crashed job with attempts remaining -> re-queued. + await store.CreateIfAbsentAsync(new JobState { JobId = "retry-me", Name = "j", Status = JobStatus.Processing, NodeId = "dead-node", LeaseExpiresUtc = expired, Attempt = 1 }, cancellationToken); + // A crashed job that exhausted its attempts -> dead-lettered. + await store.CreateIfAbsentAsync(new JobState { JobId = "give-up", Name = "j", Status = JobStatus.Processing, NodeId = "dead-node", LeaseExpiresUtc = expired, Attempt = 3 }, cancellationToken); + // A healthy job whose lease is still valid -> untouched. + await store.CreateIfAbsentAsync(new JobState { JobId = "alive", Name = "j", Status = JobStatus.Processing, NodeId = "live-node", LeaseExpiresUtc = DateTimeOffset.UtcNow.AddMinutes(5), Attempt = 1 }, cancellationToken); + // A CRON occurrence (ScheduledForUtc set) with an expired lease -> NOT reclaimed here; the scheduler owns it. + await store.CreateIfAbsentAsync(new JobState { JobId = "occurrence", Name = "j", Status = JobStatus.Processing, NodeId = "dead-node", LeaseExpiresUtc = expired, Attempt = 1, ScheduledForUtc = expired }, cancellationToken); + + int recovered = await worker.RecoverStaleAsync(maxAttempts: 3, cancellationToken: cancellationToken); + + Assert.Equal(2, recovered); + + var retried = await store.GetAsync("retry-me", cancellationToken); + Assert.Equal(JobStatus.Queued, retried!.Status); + Assert.Null(retried.NodeId); + Assert.Null(retried.LeaseExpiresUtc); + + Assert.Equal(JobStatus.DeadLettered, (await store.GetAsync("give-up", cancellationToken))!.Status); + Assert.Equal(JobStatus.Processing, (await store.GetAsync("alive", cancellationToken))!.Status); + // The CRON occurrence is left for the scheduler's own recovery, not reclaimed as a plain job. + Assert.Equal(JobStatus.Processing, (await store.GetAsync("occurrence", cancellationToken))!.Status); + } + + [Fact] + public async Task TryReclaimExpiredAsync_GuardsAgainstOwnerRenewAndForeignNodeAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var now = DateTimeOffset.UtcNow; + + await store.CreateIfAbsentAsync(new JobState { JobId = "expired", Name = "j", Status = JobStatus.Processing, NodeId = "owner", LeaseExpiresUtc = now.AddMinutes(-1), Attempt = 1 }, cancellationToken); + await store.CreateIfAbsentAsync(new JobState { JobId = "renewed", Name = "j", Status = JobStatus.Processing, NodeId = "owner", LeaseExpiresUtc = now.AddMinutes(5), Attempt = 1 }, cancellationToken); + + // Wrong owner -> rejected (another node already reclaimed/re-ran it). + Assert.False(await store.TryReclaimExpiredAsync("expired", now, "different-node", JobStatus.Queued, cancellationToken: cancellationToken)); + // Owner renewed its lease (no longer expired) -> rejected, so a live worker is never yanked out from under itself. + Assert.False(await store.TryReclaimExpiredAsync("renewed", now, "owner", JobStatus.Queued, cancellationToken: cancellationToken)); + // Still owned by the presumed-dead node and still expired -> reclaimed. + Assert.True(await store.TryReclaimExpiredAsync("expired", now, "owner", JobStatus.Queued, cancellationToken: cancellationToken)); + + Assert.Equal(JobStatus.Queued, (await store.GetAsync("expired", cancellationToken))!.Status); + Assert.Equal(JobStatus.Processing, (await store.GetAsync("renewed", cancellationToken))!.Status); + } + + [Fact] + public async Task CreateIfAbsentAsync_WithExistingJob_DoesNotOverwriteStateAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + + await store.CreateIfAbsentAsync(new JobState + { + JobId = "job-1", + Name = "first", + Status = JobStatus.Queued + }, cancellationToken); + + await store.CreateIfAbsentAsync(new JobState + { + JobId = "job-1", + Name = "second", + Status = JobStatus.Failed, + Error = "should not overwrite" + }, cancellationToken); + + var state = await store.GetAsync("job-1", cancellationToken); + + Assert.NotNull(state); + Assert.Equal("first", state.Name); + Assert.Equal(JobStatus.Queued, state.Status); + Assert.Null(state.Error); + } + + [Fact] + public async Task TryTransitionAsync_WithExpectedNodeId_RejectsStaleOwnerAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + + await store.CreateIfAbsentAsync(new JobState { JobId = "job-1", Name = "test", Status = JobStatus.Queued }, cancellationToken); + + // node-a claims and moves to Processing. + Assert.True(await store.TryTransitionAsync("job-1", JobStatus.Queued, JobStatus.Processing, new JobStatePatch { NodeId = "node-a" }, cancellationToken: cancellationToken)); + + // Its lease lapses and node-b reclaims (re-queue, then claim); node-a is no longer the owner. + Assert.True(await store.TryTransitionAsync("job-1", JobStatus.Processing, JobStatus.Queued, new JobStatePatch { ClearNodeId = true }, cancellationToken: cancellationToken)); + Assert.True(await store.TryTransitionAsync("job-1", JobStatus.Queued, JobStatus.Processing, new JobStatePatch { NodeId = "node-b" }, cancellationToken: cancellationToken)); + + // Stale node-a must NOT be able to complete the job it no longer owns (would otherwise stomp node-b's run). + Assert.False(await store.TryTransitionAsync("job-1", JobStatus.Processing, JobStatus.Completed, patch: null, expectedNodeId: "node-a", cancellationToken: cancellationToken)); + + // The current owner (node-b) can. + Assert.True(await store.TryTransitionAsync("job-1", JobStatus.Processing, JobStatus.Completed, patch: null, expectedNodeId: "node-b", cancellationToken: cancellationToken)); + + var state = await store.GetAsync("job-1", cancellationToken); + Assert.NotNull(state); + Assert.Equal(JobStatus.Completed, state.Status); + } + + [Fact] + public async Task TryClaimAsync_WhenLeaseIsHeldByAnotherNode_ReturnsFalseUntilLeaseExpiresAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + + await store.CreateIfAbsentAsync(new JobState + { + JobId = "job-1", + Name = "test", + Status = JobStatus.Queued + }, cancellationToken); + + Assert.True(await store.TryClaimAsync("job-1", "node-a", TimeSpan.FromMinutes(1), cancellationToken)); + Assert.False(await store.TryClaimAsync("job-1", "node-b", TimeSpan.FromMinutes(1), cancellationToken)); + Assert.True(await store.ReleaseClaimAsync("job-1", "node-a", cancellationToken)); + Assert.True(await store.TryClaimAsync("job-1", "node-b", TimeSpan.FromMinutes(1), cancellationToken)); + + var state = await store.GetAsync("job-1", cancellationToken); + Assert.NotNull(state); + Assert.Equal("node-b", state.NodeId); + Assert.NotNull(state.LeaseExpiresUtc); + } + + [Fact] + public async Task ClaimDueDispatchesAsync_ClaimsReleasesAndCompletesDueDispatchesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var now = DateTimeOffset.UtcNow; + + await store.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = "dispatch-1", + Kind = ScheduledDispatchKind.QueueMessage, + Destination = DestinationAddress.ForQueue("work"), + Body = "hello"u8.ToArray(), + DueUtc = now.AddSeconds(-1) + }, cancellationToken); + + await store.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = "dispatch-2", + Kind = ScheduledDispatchKind.QueueMessage, + Destination = DestinationAddress.ForQueue("work"), + Body = "later"u8.ToArray(), + DueUtc = now.AddHours(1) + }, cancellationToken); + + var claimed = await store.ClaimDueDispatchesAsync(now, 10, "node-a", TimeSpan.FromMinutes(1), cancellationToken); + + var dispatch = Assert.Single(claimed); + Assert.Equal("dispatch-1", dispatch.DispatchId); + Assert.Equal("node-a", dispatch.ClaimOwner); + Assert.Equal(1, dispatch.Attempts); + + var claimedAgain = await store.ClaimDueDispatchesAsync(now, 10, "node-b", TimeSpan.FromMinutes(1), cancellationToken); + Assert.Empty(claimedAgain); + + await store.ReleaseDispatchAsync("dispatch-1", "node-a", now.AddSeconds(-1), cancellationToken); + + var reclaimed = await store.ClaimDueDispatchesAsync(now, 10, "node-b", TimeSpan.FromMinutes(1), cancellationToken); + dispatch = Assert.Single(reclaimed); + Assert.Equal("node-b", dispatch.ClaimOwner); + Assert.Equal(2, dispatch.Attempts); + + await store.CompleteDispatchAsync("dispatch-1", "node-b", cancellationToken); + + var afterComplete = await store.ClaimDueDispatchesAsync(now, 10, "node-c", TimeSpan.FromMinutes(1), cancellationToken); + Assert.Empty(afterComplete); + } + + [Fact] + public async Task RunAsync_WhenJobSucceeds_TracksCompletedStateAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var probe = new JobRuntimeProbe(); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(probe) + .BuildServiceProvider(); + var client = new JobClient(store); + var worker = new JobWorker(store, serviceProvider, nodeId: "node-a"); + + JobHandle handle = await client.EnqueueAsync(new JobRequestOptions { JobId = "job-1" }, cancellationToken); + Assert.True(await worker.RunAsync(handle.JobId, cancellationToken)); + + var state = await handle.GetStateAsync(cancellationToken); + Assert.NotNull(state); + Assert.Equal(1, probe.RunCount); + Assert.Equal(JobStatus.Completed, state.Status); + Assert.Equal(1, state.Attempt); + Assert.Equal(100, state.Progress); + Assert.NotNull(state.StartedUtc); + Assert.NotNull(state.CompletedUtc); + Assert.Null(state.NodeId); + Assert.Null(state.LeaseExpiresUtc); + } + + [Fact] + public async Task EnqueueAsync_WithRegisteredJobType_PersistsStableNameAndWorkerResolvesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var probe = new JobRuntimeProbe(); + var registry = new JobTypeRegistry([new JobTypeRegistration("search.rebuild", typeof(SuccessfulTrackedJob))]); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(probe) + .BuildServiceProvider(); + var client = new JobClient(store, jobTypes: registry); + var worker = new JobWorker(store, serviceProvider, nodeId: "node-a", jobTypes: registry); + + JobHandle handle = await client.EnqueueAsync(new JobRequestOptions { JobId = "job-registered" }, cancellationToken); + var queued = await handle.GetStateAsync(cancellationToken); + + Assert.NotNull(queued); + Assert.Equal("search.rebuild", queued.JobType); + Assert.DoesNotContain(",", queued.JobType); + Assert.True(await worker.RunAsync(handle.JobId, cancellationToken)); + + var completed = await handle.GetStateAsync(cancellationToken); + Assert.NotNull(completed); + Assert.Equal(JobStatus.Completed, completed.Status); + Assert.Equal(1, probe.RunCount); + } + + [Fact] + public async Task RequestCancellationAsync_WhenJobIsRunning_CancelsAndTracksStateAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var probe = new JobRuntimeProbe(); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(probe) + .BuildServiceProvider(); + var client = new JobClient(store); + var worker = new JobWorker(store, serviceProvider, nodeId: "node-a"); + + JobHandle handle = await client.EnqueueAsync(new JobRequestOptions { JobId = "job-1" }, cancellationToken); + var runTask = worker.RunAsync(handle.JobId, cancellationToken); + await probe.Started.Task.WaitAsync(TimeSpan.FromSeconds(5), cancellationToken); + + Assert.True(await handle.RequestCancellationAsync(cancellationToken)); + + await probe.Cancelled.Task.WaitAsync(TimeSpan.FromSeconds(5), cancellationToken); + Assert.True(await runTask.WaitAsync(TimeSpan.FromSeconds(5), cancellationToken)); + var state = await handle.GetStateAsync(cancellationToken); + + Assert.NotNull(state); + Assert.Equal(JobStatus.Cancelled, state.Status); + Assert.True(state.CancellationRequested); + Assert.NotNull(state.CompletedUtc); + Assert.Null(state.NodeId); + Assert.Null(state.LeaseExpiresUtc); + } + + private sealed class JobRuntimeProbe + { + private int _runCount; + + public TaskCompletionSource Started { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + public TaskCompletionSource Cancelled { get; } = new(TaskCreationOptions.RunContinuationsAsynchronously); + + public int RunCount => Volatile.Read(ref _runCount); + public string? LastMessage { get; private set; } + + public void RecordRun(string? message = null) + { + Interlocked.Increment(ref _runCount); + LastMessage = message; + } + } + + [Fact] + public async Task EnqueueAsync_WithTypedArguments_JobReceivesDeserializedPayloadAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var probe = new JobRuntimeProbe(); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(probe) + .BuildServiceProvider(); + var client = new JobClient(store); + var worker = new JobWorker(store, serviceProvider, nodeId: "node-a"); + + var handle = await client.EnqueueAsync(new ResizeArgs { Path = "/img/1.png", Width = 640 }, cancellationToken: cancellationToken); + + // The payload and its discriminator are durable state, not in-process context. + var state = await store.GetAsync(handle.JobId, cancellationToken); + Assert.NotNull(state?.Payload); + Assert.Equal(typeof(ResizeArgs).FullName, state.PayloadType); + + Assert.True(await worker.RunAsync(handle.JobId, cancellationToken)); + Assert.Equal("/img/1.png:640", probe.LastMessage); + Assert.Equal(JobStatus.Completed, (await handle.GetStateAsync(cancellationToken))!.Status); + } + + [Fact] + public async Task GetArguments_WhenEnqueuedWithout_ThrowsDescriptiveErrorAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var probe = new JobRuntimeProbe(); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(probe) + .BuildServiceProvider(); + var client = new JobClient(store); + var worker = new JobWorker(store, serviceProvider, nodeId: "node-a"); + + // The args-requiring job was enqueued via the argless API: the run fails (job faults) rather than silently + // executing with defaults, and the error names the fix. + var handle = await client.EnqueueAsync(cancellationToken: cancellationToken); + Assert.True(await worker.RunAsync(handle.JobId, cancellationToken)); + + var state = await handle.GetStateAsync(cancellationToken); + Assert.Equal(JobStatus.Failed, state!.Status); + Assert.Contains("without arguments", state.Error); + } + + [Fact] + public async Task RunJob_ResolvesScopedServicesPerExecutionAndDisposesThemAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var tracker = new ScopedLifetimeTracker(); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(tracker) + .AddScoped() + .AddTransient() + .BuildServiceProvider(); + var client = new JobClient(store); + var worker = new JobWorker(store, serviceProvider, nodeId: "node-a"); + + var first = await client.EnqueueAsync(cancellationToken: cancellationToken); + var second = await client.EnqueueAsync(cancellationToken: cancellationToken); + Assert.True(await worker.RunAsync(first.JobId, cancellationToken)); + Assert.True(await worker.RunAsync(second.JobId, cancellationToken)); + + // Two runs -> two scoped instances (not one root-container singleton), each disposed when its run ended. + Assert.Equal(2, tracker.Created); + Assert.Equal(2, tracker.Disposed); + } + + [Fact] + public async Task RunQueuedAsync_WithMaxConcurrency_RespectsCapAndRunsInParallelAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var gauge = new ConcurrencyGauge(); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(gauge) + .BuildServiceProvider(); + var client = new JobClient(store); + var worker = new JobWorker(store, serviceProvider, nodeId: "node-a", maxConcurrency: 2); + + for (int i = 0; i < 6; i++) + await client.EnqueueAsync(cancellationToken: cancellationToken); + + Assert.Equal(6, await worker.RunQueuedAsync(cancellationToken: cancellationToken)); + Assert.True(gauge.MaxObserved <= 2, $"expected at most 2 in-flight jobs, observed {gauge.MaxObserved}"); + Assert.True(gauge.MaxObserved > 1, "expected the pool to actually run jobs in parallel"); + } + + private sealed class ScopedLifetimeTracker + { + private int _created; + private int _disposed; + public int Created => Volatile.Read(ref _created); + public int Disposed => Volatile.Read(ref _disposed); + public void RecordCreated() => Interlocked.Increment(ref _created); + public void RecordDisposed() => Interlocked.Increment(ref _disposed); + } + + private sealed class ScopedDependency : IDisposable + { + private readonly ScopedLifetimeTracker _tracker; + + public ScopedDependency(ScopedLifetimeTracker tracker) + { + _tracker = tracker; + _tracker.RecordCreated(); + } + + public void Dispose() => _tracker.RecordDisposed(); + } + + private sealed class ScopedConsumingJob : IJob + { + // The dependency's usefulness is its lifetime tracking; resolving it is the test. + public ScopedConsumingJob(ScopedDependency dependency) => _ = dependency; + + public Task RunAsync(JobExecutionContext context) => Task.FromResult(JobResult.Success); + } + + private sealed class ConcurrencyGauge + { + private int _inFlight; + private int _maxObserved; + + public int MaxObserved => Volatile.Read(ref _maxObserved); + + public async Task TrackAsync() + { + int current = Interlocked.Increment(ref _inFlight); + int max; + while (current > (max = Volatile.Read(ref _maxObserved))) + Interlocked.CompareExchange(ref _maxObserved, current, max); + + await Task.Delay(TimeSpan.FromMilliseconds(100)); + Interlocked.Decrement(ref _inFlight); + } + } + + private sealed class ConcurrencyProbeJob : IJob + { + private readonly ConcurrencyGauge _gauge; + + public ConcurrencyProbeJob(ConcurrencyGauge gauge) => _gauge = gauge; + + public async Task RunAsync(JobExecutionContext context) + { + await _gauge.TrackAsync(); + return JobResult.Success; + } + } + + private sealed class ResizeArgs + { + public string? Path { get; set; } + public int Width { get; set; } + } + + private sealed class ArgsConsumingJob : IJob + { + private readonly JobRuntimeProbe _probe; + + public ArgsConsumingJob(JobRuntimeProbe probe) + { + _probe = probe; + } + + public Task RunAsync(JobExecutionContext context) + { + var args = context.GetArguments(); + _probe.RecordRun($"{args.Path}:{args.Width}"); + return Task.FromResult(JobResult.Success); + } + } + + private sealed class SuccessfulTrackedJob : IJob + { + private readonly JobRuntimeProbe _probe; + + public SuccessfulTrackedJob(JobRuntimeProbe probe) + { + _probe = probe; + } + + public Task RunAsync(JobExecutionContext context) + { + context.CancellationToken.ThrowIfCancellationRequested(); + _probe.RecordRun(); + return Task.FromResult(JobResult.Success); + } + } + + private sealed class CancellableTrackedJob : IJob + { + private readonly JobRuntimeProbe _probe; + + public CancellableTrackedJob(JobRuntimeProbe probe) + { + _probe = probe; + } + + public async Task RunAsync(JobExecutionContext context) + { + _probe.Started.TrySetResult(); + + try + { + await Task.Delay(TimeSpan.FromMinutes(1), context.CancellationToken); + return JobResult.Success; + } + catch (OperationCanceledException) + { + _probe.Cancelled.TrySetResult(); + throw; + } + } + } + + private sealed class ProgressJob : IJob + { + public async Task RunAsync(JobExecutionContext context) + { + await context.ReportProgressAsync(75, $"{context.JobId}:{context.Attempt}", context.CancellationToken); + return JobResult.Success; + } + } +} diff --git a/tests/Foundatio.Tests/Jobs/JobSchedulerTests.cs b/tests/Foundatio.Tests/Jobs/JobSchedulerTests.cs new file mode 100644 index 000000000..4b8fdafec --- /dev/null +++ b/tests/Foundatio.Tests/Jobs/JobSchedulerTests.cs @@ -0,0 +1,518 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Xunit; + +namespace Foundatio.Tests.Jobs; + +public class JobSchedulerTests +{ + [Fact] + public async Task EnqueueDueOccurrencesAsync_WhenOccurrenceIsDue_CreatesSingleGlobalOccurrenceAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryJobScheduler(); + var store = new InMemoryJobRuntimeStore(); + var processor = CreateProcessor(scheduler, store, "node-a"); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "nightly", + Cron = "* * * * *", + JobType = typeof(ScheduledProbeJob) + }, cancellationToken); + + var first = await processor.EnqueueDueOccurrencesAsync(now, cancellationToken); + var second = await processor.EnqueueDueOccurrencesAsync(now, cancellationToken); + + var dispatch = Assert.Single(first); + Assert.Empty(second); + Assert.Equal("nightly:20260101000000:global", dispatch.DispatchId); + Assert.Equal(ScheduledDispatchKind.JobOccurrence, dispatch.Kind); + Assert.Equal("nightly", dispatch.Headers["job.name"]); + + var state = await store.GetAsync(dispatch.JobId!, cancellationToken); + Assert.NotNull(state); + Assert.Equal(JobStatus.Scheduled, state.Status); + Assert.Equal(new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero), state.ScheduledForUtc); + } + + [Fact] + public async Task EnqueueDueOccurrencesAsync_WithAllowConcurrent_MaterializesEveryMissedOccurrenceAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryJobScheduler(); + var store = new InMemoryJobRuntimeStore(); + var processor = CreateProcessor(scheduler, store, "node-a"); + var now = new DateTimeOffset(2026, 1, 1, 0, 5, 30, TimeSpan.Zero); + + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "frequent", + Cron = "* * * * *", + JobType = typeof(ScheduledProbeJob), + Overlap = OverlapPolicy.AllowConcurrent, + MisfireWindow = TimeSpan.FromMinutes(10) + }, cancellationToken); + + // A scheduler that lagged behind a per-minute cadence must materialize every missed occurrence in the window, + // not just the most recent one. + var first = await processor.EnqueueDueOccurrencesAsync(now, cancellationToken); + Assert.True(first.Count >= 5, $"Expected multiple missed occurrences, got {first.Count}"); + + // Deterministic occurrence ids dedupe across overlapping windows: a second pass at the same time adds nothing. + var second = await processor.EnqueueDueOccurrencesAsync(now, cancellationToken); + Assert.Empty(second); + } + + [Fact] + public async Task JobRuntimeService_RunsQueuedJobsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var probe = new JobSchedulerProbe(); + var serviceProvider = new ServiceCollection().AddSingleton(probe).BuildServiceProvider(); + var store = new InMemoryJobRuntimeStore(); + var scheduler = new InMemoryJobScheduler(); + var registry = new JobTypeRegistry([new JobTypeRegistration("probe", typeof(ScheduledProbeJob))]); + var worker = new JobWorker(store, serviceProvider, nodeId: "node-a", jobTypes: registry); + var processor = new JobScheduleProcessor(scheduler, store, worker, nodeId: "node-a", jobTypes: registry); + var client = new JobClient(store, jobTypes: registry); + + var service = new Foundatio.Extensions.Hosting.Jobs.JobRuntimeService(processor, worker, + options: new Foundatio.Extensions.Hosting.Jobs.JobRuntimeServiceOptions { PollInterval = TimeSpan.FromMilliseconds(50) }); + + await ((Microsoft.Extensions.Hosting.IHostedService)service).StartAsync(cancellationToken); + try + { + var handle = await client.EnqueueAsync(cancellationToken: cancellationToken); + + JobState? state = null; + for (int i = 0; i < 100 && (state = await handle.GetStateAsync(cancellationToken))?.Status != JobStatus.Completed; i++) + await Task.Delay(50, cancellationToken); + + Assert.Equal(JobStatus.Completed, state?.Status); + Assert.Equal(1, probe.RunCount); + } + finally + { + await ((Microsoft.Extensions.Hosting.IHostedService)service).StopAsync(cancellationToken); + } + } + + [Fact] + public async Task RunDueOccurrencesAsync_WhenOccurrenceIsDue_RunsConfiguredJobAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryJobScheduler(); + var store = new InMemoryJobRuntimeStore(); + var probe = new JobSchedulerProbe(); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(probe) + .BuildServiceProvider(); + var worker = new JobWorker(store, serviceProvider, nodeId: "node-a"); + var processor = new JobScheduleProcessor(scheduler, store, worker, nodeId: "node-a"); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "nightly", + Cron = "* * * * *", + JobType = typeof(ScheduledProbeJob) + }, cancellationToken); + var scheduled = await processor.EnqueueDueOccurrencesAsync(now, cancellationToken); + + int completed = await processor.RunDueOccurrencesAsync(now, cancellationToken: cancellationToken); + + var dispatch = Assert.Single(scheduled); + var state = await store.GetAsync(dispatch.JobId!, cancellationToken); + Assert.Equal(1, completed); + Assert.Equal(1, probe.RunCount); + Assert.NotNull(state); + Assert.Equal(JobStatus.Completed, state.Status); + Assert.Equal(1, state.Attempt); + Assert.Equal(100, state.Progress); + } + + [Fact] + public async Task EnqueueDueOccurrencesAsync_WithPerNodeScope_CreatesOccurrencePerNodeAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryJobScheduler(); + var store = new InMemoryJobRuntimeStore(); + var nodeA = CreateProcessor(scheduler, store, "node-a"); + var nodeB = CreateProcessor(scheduler, store, "node-b"); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "per-node", + Cron = "* * * * *", + JobType = typeof(ScheduledProbeJob), + Scope = ScheduledJobScope.PerNode + }, cancellationToken); + + var first = await nodeA.EnqueueDueOccurrencesAsync(now, cancellationToken); + var second = await nodeB.EnqueueDueOccurrencesAsync(now, cancellationToken); + + Assert.Equal("per-node:20260101000000:node-a", Assert.Single(first).DispatchId); + Assert.Equal("per-node:20260101000000:node-b", Assert.Single(second).DispatchId); + + var states = await store.QueryAsync(new JobQuery { Name = "per-node" }, cancellationToken); + Assert.Equal(2, states.Count); + } + + [Fact] + public async Task EnqueueDueOccurrencesAsync_WithMisfireWindow_CatchesRecentMissedOccurrenceAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryJobScheduler(); + var store = new InMemoryJobRuntimeStore(); + var processor = CreateProcessor(scheduler, store, "node-a"); + var now = new DateTimeOffset(2026, 1, 1, 0, 5, 0, TimeSpan.Zero); + + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "daily", + Cron = "0 0 * * *", + JobType = typeof(ScheduledProbeJob), + MisfireWindow = TimeSpan.FromMinutes(10) + }, cancellationToken); + + var scheduled = await processor.EnqueueDueOccurrencesAsync(now, cancellationToken); + + var dispatch = Assert.Single(scheduled); + var state = await store.GetAsync(dispatch.JobId!, cancellationToken); + Assert.NotNull(state); + Assert.Equal(new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero), state.ScheduledForUtc); + } + + [Fact] + public async Task RunDueOccurrencesAsync_WhenDispatchIsQueueMessage_MaterializesItAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryJobScheduler(); + var store = new InMemoryJobRuntimeStore(); + await using var transport = new InMemoryMessageTransport(); + var processor = CreateProcessor(scheduler, store, "node-a", transport); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero); + + await store.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = "delayed-message", + Kind = ScheduledDispatchKind.QueueMessage, + Destination = DestinationAddress.ForQueue("work"), + Body = "hello"u8.ToArray(), + DueUtc = now + }, cancellationToken); + + int completed = await processor.RunDueOccurrencesAsync(now, cancellationToken: cancellationToken); + + Assert.Equal(1, completed); + var pull = Assert.IsAssignableFrom(transport); + var entries = await pull.ReceiveAsync(DestinationAddress.ForQueue("work"), new ReceiveRequest { MaxMessages = 1, MaxWaitTime = TimeSpan.FromMilliseconds(50) }, cancellationToken); + var entry = Assert.Single(entries); + Assert.Equal("delayed-message", entry.Id); + Assert.Equal("hello"u8.ToArray(), entry.Body.ToArray()); + } + + [Fact] + public async Task RunDueOccurrencesAsync_WhenJobFails_RetriesThenDeadLettersAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryJobScheduler(); + var store = new InMemoryJobRuntimeStore(); + var probe = new JobSchedulerProbe(); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(probe) + .BuildServiceProvider(); + var worker = new JobWorker(store, serviceProvider, nodeId: "node-a"); + var processor = new JobScheduleProcessor(scheduler, store, worker, nodeId: "node-a"); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "nightly", + Cron = "* * * * *", + JobType = typeof(FailingScheduledJob), + MaxRetries = 1 + }, cancellationToken); + var scheduled = await processor.EnqueueDueOccurrencesAsync(now, cancellationToken); + var dispatch = Assert.Single(scheduled); + + Assert.Equal(0, await processor.RunDueOccurrencesAsync(now, cancellationToken: cancellationToken)); + var retried = await store.GetAsync(dispatch.JobId!, cancellationToken); + Assert.NotNull(retried); + Assert.Equal(JobStatus.Scheduled, retried.Status); + Assert.Equal(1, retried.Attempt); + + Assert.Equal(1, await processor.RunDueOccurrencesAsync(now.AddMinutes(2), cancellationToken: cancellationToken)); + var deadlettered = await store.GetAsync(dispatch.JobId!, cancellationToken); + Assert.NotNull(deadlettered); + Assert.Equal(JobStatus.DeadLettered, deadlettered.Status); + Assert.Equal(2, deadlettered.Attempt); + Assert.Equal(2, probe.RunCount); + } + + [Fact] + public async Task RunDueOccurrencesAsync_WhenProcessingLeaseExpired_ReclaimsAndRunsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryJobScheduler(); + var store = new InMemoryJobRuntimeStore(); + var probe = new JobSchedulerProbe(); + await using var serviceProvider = new ServiceCollection() + .AddSingleton(probe) + .BuildServiceProvider(); + var worker = new JobWorker(store, serviceProvider, nodeId: "node-a"); + var processor = new JobScheduleProcessor(scheduler, store, worker, nodeId: "node-a"); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + const string jobId = "nightly:20260101000000:global"; + + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "nightly", + Cron = "* * * * *", + JobType = typeof(ScheduledProbeJob), + MaxRetries = 1 + }, cancellationToken); + await store.CreateIfAbsentAsync(new JobState + { + JobId = jobId, + Name = "nightly", + Status = JobStatus.Processing, + Attempt = 1, + NodeId = "node-b", + LeaseExpiresUtc = now.AddMinutes(-1), + ScheduledForUtc = now.AddSeconds(-30) + }, cancellationToken); + await store.ScheduleDispatchAsync(new ScheduledDispatchState + { + DispatchId = jobId, + Kind = ScheduledDispatchKind.JobOccurrence, + JobName = "nightly", + Body = Array.Empty(), + DueUtc = now, + JobId = jobId + }, cancellationToken); + + Assert.Equal(1, await processor.RunDueOccurrencesAsync(now, cancellationToken: cancellationToken)); + + var state = await store.GetAsync(jobId, cancellationToken); + Assert.NotNull(state); + Assert.Equal(JobStatus.Completed, state.Status); + Assert.Equal(2, state.Attempt); + Assert.Equal(1, probe.RunCount); + } + + [Fact] + public async Task RunQueuedAsync_DoesNotClaimScheduledOccurrencesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + var probe = new JobSchedulerProbe(); + await using var serviceProvider = new ServiceCollection().AddSingleton(probe).BuildServiceProvider(); + var worker = new JobWorker(store, serviceProvider, nodeId: "node-a"); + + // A CRON occurrence sitting in Queued (the scheduler transitioned it Scheduled->Queued) must NOT be claimed by + // the generic worker — only the scheduler runs occurrences, with its own retry/dead-letter accounting. + await store.CreateIfAbsentAsync(new JobState + { + JobId = "nightly:20260101000000:global", + Name = "nightly", + JobType = typeof(ScheduledProbeJob).FullName, + Status = JobStatus.Queued, + ScheduledForUtc = new DateTimeOffset(2026, 1, 1, 0, 0, 0, TimeSpan.Zero) + }, cancellationToken); + + Assert.Equal(0, await worker.RunQueuedAsync(cancellationToken: cancellationToken)); + Assert.Equal(0, probe.RunCount); + Assert.Equal(JobStatus.Queued, (await store.GetAsync("nightly:20260101000000:global", cancellationToken))!.Status); + } + + [Fact] + public async Task RunDueOccurrencesAsync_WhenOccurrenceIsTerminal_RetiresDispatchInsteadOfReschedulingAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryJobScheduler(); + var store = new InMemoryJobRuntimeStore(); + var processor = CreateProcessor(scheduler, store, "node-a"); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + const string jobId = "nightly:20260101000000:global"; + + await scheduler.ScheduleAsync(new ScheduledJobDefinition { Name = "nightly", Cron = "* * * * *", JobType = typeof(ScheduledProbeJob) }, cancellationToken); + // A worker completed the occurrence but crashed before retiring its dispatch: a terminal job with a live dispatch. + await store.CreateIfAbsentAsync(new JobState { JobId = jobId, Name = "nightly", Status = JobStatus.Completed, ScheduledForUtc = now.AddSeconds(-30) }, cancellationToken); + await store.ScheduleDispatchAsync(new ScheduledDispatchState { DispatchId = jobId, Kind = ScheduledDispatchKind.JobOccurrence, JobName = "nightly", Body = Array.Empty(), DueUtc = now, JobId = jobId }, cancellationToken); + + await processor.RunDueOccurrencesAsync(now, cancellationToken: cancellationToken); + + // The dispatch for a terminal occurrence must be retired, not rescheduled +1min and re-claimed forever. + Assert.Empty(await store.ClaimDueDispatchesAsync(now.AddMinutes(5), 10, "node-b", TimeSpan.FromMinutes(5), cancellationToken)); + } + + [Fact] + public async Task EnqueueDueOccurrencesAsync_PerNodeScope_WithDelimiterInNodeId_DoesNotCrossMatchAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var scheduler = new InMemoryJobScheduler(); + var store = new InMemoryJobRuntimeStore(); + // Node ids that are suffix-confusable under a naive EndsWith(":{scope}") check — the default NodeIdentity contains ':'. + var nodeXB = CreateProcessor(scheduler, store, "x:b"); + var nodeB = CreateProcessor(scheduler, store, "b"); + var now = new DateTimeOffset(2026, 1, 1, 0, 0, 30, TimeSpan.Zero); + + await scheduler.ScheduleAsync(new ScheduledJobDefinition + { + Name = "per-node", + Cron = "* * * * *", + JobType = typeof(ScheduledProbeJob), + Scope = ScheduledJobScope.PerNode // default Overlap = SkipIfRunning, which runs the active-occurrence check + }, cancellationToken); + + Assert.Single(await nodeXB.EnqueueDueOccurrencesAsync(now, cancellationToken)); // creates "per-node:...:x:b" + // node "b" must still materialize its own occurrence; node "x:b"'s occurrence must not be mistaken for node "b"'s. + Assert.Single(await nodeB.EnqueueDueOccurrencesAsync(now, cancellationToken)); // creates "per-node:...:b" + + var states = await store.QueryAsync(new JobQuery { Name = "per-node", Limit = 100 }, cancellationToken); + Assert.Equal(2, states.Count); + } + + [Fact] + public async Task AddFoundatio_WithRuntimeStore_AutoRegistersAndRunsPumpAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var probe = new JobSchedulerProbe(); + var services = new ServiceCollection().AddSingleton(probe); + var foundatio = services.AddFoundatio(); + foundatio.Jobs.UseInMemoryRuntime(); + foundatio.Jobs.Register("probe"); + await using var provider = services.BuildServiceProvider(); + + // Configuring a runtime store auto-registers the pump — no separate AddJobRuntimeService — so a hosted process + // runs IJobClient-submitted jobs (and drains delayed messaging) without extra wiring. + var pump = Assert.Single(provider.GetServices().OfType()); + await pump.StartAsync(cancellationToken); + try + { + var handle = await provider.GetRequiredService().EnqueueAsync(cancellationToken: cancellationToken); + + JobState? state = null; + for (int i = 0; i < 100 && (state = await handle.GetStateAsync(cancellationToken))?.Status != JobStatus.Completed; i++) + await Task.Delay(50, cancellationToken); + + Assert.Equal(JobStatus.Completed, state?.Status); + Assert.Equal(1, probe.RunCount); + } + finally + { + await pump.StopAsync(cancellationToken); + } + } + + [Fact] + public async Task ConfigureRuntimePump_Disabled_DoesNotPumpAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var probe = new JobSchedulerProbe(); + var services = new ServiceCollection().AddSingleton(probe); + var foundatio = services.AddFoundatio(); + foundatio.Jobs.UseInMemoryRuntime(); + foundatio.Jobs.Register("probe"); + foundatio.Jobs.ConfigureRuntimePump(o => o.Enabled = false); // opt out of automatic pumping + await using var provider = services.BuildServiceProvider(); + + var pump = Assert.Single(provider.GetServices().OfType()); + await pump.StartAsync(cancellationToken); + try + { + var handle = await provider.GetRequiredService().EnqueueAsync(cancellationToken: cancellationToken); + + // With the pump disabled, the job is never claimed: it stays Queued and the job never runs. + await Task.Delay(300, cancellationToken); + Assert.Equal(JobStatus.Queued, (await handle.GetStateAsync(cancellationToken))!.Status); + Assert.Equal(0, probe.RunCount); + } + finally + { + await pump.StopAsync(cancellationToken); + } + } + + [Fact] + public async Task AddJobRuntimeService_BeforeUseRuntimeStore_RegistersExactlyOnePumpAsync() + { + var services = new ServiceCollection().AddSingleton(new JobSchedulerProbe()); + // Hosting-first ordering must not stack a second pump: AddJobRuntimeService only tunes the single core pump. + Foundatio.Extensions.Hosting.Jobs.JobHostExtensions.AddJobRuntimeService(services, o => o.PollInterval = TimeSpan.FromMilliseconds(25)); + services.AddFoundatio().Jobs.UseInMemoryRuntime(); + await using var provider = services.BuildServiceProvider(); + + var hostedServices = provider.GetServices().ToList(); + Assert.Single(hostedServices.OfType()); + Assert.Empty(hostedServices.OfType()); + // The options passed to AddJobRuntimeService are carried onto that single pump. + Assert.Equal(TimeSpan.FromMilliseconds(25), provider.GetRequiredService().PollInterval); + } + + private static JobScheduleProcessor CreateProcessor(IJobScheduler scheduler, IJobRuntimeStore store, string nodeId, IMessageTransport? transport = null) + { + var serviceProvider = new ServiceCollection() + .AddSingleton(new JobSchedulerProbe()) + .BuildServiceProvider(); + var worker = new JobWorker(store, serviceProvider, nodeId: nodeId); + return new JobScheduleProcessor(scheduler, store, worker, nodeId: nodeId, transport: transport); + } + + private sealed class JobSchedulerProbe + { + private int _runCount; + + public int RunCount => Volatile.Read(ref _runCount); + + public void RecordRun() + { + Interlocked.Increment(ref _runCount); + } + } + + private sealed class FailingScheduledJob : IJob + { + private readonly JobSchedulerProbe _probe; + + public FailingScheduledJob(JobSchedulerProbe probe) + { + _probe = probe; + } + + public Task RunAsync(JobExecutionContext context) + { + context.CancellationToken.ThrowIfCancellationRequested(); + _probe.RecordRun(); + return Task.FromResult(JobResult.FromException(new InvalidOperationException("failed"))); + } + } + + private sealed class ScheduledProbeJob : IJob + { + private readonly JobSchedulerProbe _probe; + + public ScheduledProbeJob(JobSchedulerProbe probe) + { + _probe = probe; + } + + public Task RunAsync(JobExecutionContext context) + { + context.CancellationToken.ThrowIfCancellationRequested(); + _probe.RecordRun(); + return Task.FromResult(JobResult.Success); + } + } +} diff --git a/tests/Foundatio.Tests/Jobs/JobTests.cs b/tests/Foundatio.Tests/Jobs/JobTests.cs index 93e12c694..94eb34faa 100644 --- a/tests/Foundatio.Tests/Jobs/JobTests.cs +++ b/tests/Foundatio.Tests/Jobs/JobTests.cs @@ -5,7 +5,7 @@ using System.Threading; using System.Threading.Tasks; using Foundatio.Caching; -using Foundatio.Jobs; +using Foundatio.Jobs.Legacy; using Foundatio.Xunit; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging; diff --git a/tests/Foundatio.Tests/Jobs/LeaseSupervisionTests.cs b/tests/Foundatio.Tests/Jobs/LeaseSupervisionTests.cs new file mode 100644 index 000000000..61bef643f --- /dev/null +++ b/tests/Foundatio.Tests/Jobs/LeaseSupervisionTests.cs @@ -0,0 +1,101 @@ +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Jobs; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +namespace Foundatio.Tests.Jobs; + +public class LeaseSupervisionTests +{ + [Fact] + public async Task RenewalDenied_CancelsRunningJobAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new LeaseFailingStore(new InMemoryJobRuntimeStore()) { DenyRenewals = true }; + await using var serviceProvider = new ServiceCollection().BuildServiceProvider(); + var client = new JobClient(store); + var worker = new JobWorker(store, serviceProvider, nodeId: "node-a", lease: TimeSpan.FromSeconds(1)); + + var handle = await client.EnqueueAsync(cancellationToken: cancellationToken); + Assert.True(await worker.RunAsync(handle.JobId, cancellationToken)); + + // A clean "renewal denied" means another node owns the lease: the run must be cancelled, not left executing. + var state = await handle.GetStateAsync(cancellationToken); + Assert.Equal(JobStatus.Cancelled, state!.Status); + } + + [Fact] + public async Task RenewalThrowingPastLeaseWindow_CancelsRunningJobAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new LeaseFailingStore(new InMemoryJobRuntimeStore()) { ThrowOnRenewals = true }; + await using var serviceProvider = new ServiceCollection().BuildServiceProvider(); + var client = new JobClient(store); + var worker = new JobWorker(store, serviceProvider, nodeId: "node-a", lease: TimeSpan.FromSeconds(1)); + + var handle = await client.EnqueueAsync(cancellationToken: cancellationToken); + Assert.True(await worker.RunAsync(handle.JobId, cancellationToken)); + + // Renewal that keeps THROWING must not let the run outlive its lease: once the window passes without one + // successful renewal, another node may have reclaimed the job, so continuing would double-run side effects. + var state = await handle.GetStateAsync(cancellationToken); + Assert.Equal(JobStatus.Cancelled, state!.Status); + } + + private sealed class WaitForCancellationJob : IJob + { + public async Task RunAsync(JobExecutionContext context) + { + try + { + // Runs "forever" unless the supervision loop cancels the run. + await Task.Delay(TimeSpan.FromSeconds(30), context.CancellationToken); + } + catch (OperationCanceledException) + { + return JobResult.CancelledWithMessage("lease lost"); + } + + return JobResult.FailedWithMessage("was never cancelled"); + } + } + + // Delegates everything to the inner store; renewals can be denied (clean lease loss) or made to throw (store outage). + private sealed class LeaseFailingStore : IJobRuntimeStore + { + private readonly IJobRuntimeStore _inner; + + public LeaseFailingStore(IJobRuntimeStore inner) => _inner = inner; + + public bool DenyRenewals { get; set; } + public bool ThrowOnRenewals { get; set; } + + public Task RenewClaimAsync(string jobId, string nodeId, TimeSpan lease, CancellationToken cancellationToken = default) + { + if (ThrowOnRenewals) + throw new TimeoutException("store unreachable"); + + return DenyRenewals ? Task.FromResult(false) : _inner.RenewClaimAsync(jobId, nodeId, lease, cancellationToken); + } + + public Task GetAsync(string jobId, CancellationToken ct = default) => _inner.GetAsync(jobId, ct); + public Task> QueryAsync(JobQuery query, CancellationToken ct = default) => _inner.QueryAsync(query, ct); + public Task CreateIfAbsentAsync(JobState initial, CancellationToken ct = default) => _inner.CreateIfAbsentAsync(initial, ct); + public Task TryTransitionAsync(string jobId, JobStatus expectedStatus, JobStatus newStatus, JobStatePatch? patch = null, string? expectedNodeId = null, CancellationToken ct = default) => _inner.TryTransitionAsync(jobId, expectedStatus, newStatus, patch, expectedNodeId, ct); + public Task TryClaimAsync(string jobId, string nodeId, TimeSpan lease, CancellationToken ct = default) => _inner.TryClaimAsync(jobId, nodeId, lease, ct); + public Task ReleaseClaimAsync(string jobId, string nodeId, CancellationToken ct = default) => _inner.ReleaseClaimAsync(jobId, nodeId, ct); + public Task> GetExpiredProcessingAsync(DateTimeOffset now, int limit, CancellationToken ct = default) => _inner.GetExpiredProcessingAsync(now, limit, ct); + public Task TryReclaimExpiredAsync(string jobId, DateTimeOffset now, string expectedNodeId, JobStatus newStatus, JobStatePatch? patch = null, CancellationToken ct = default) => _inner.TryReclaimExpiredAsync(jobId, now, expectedNodeId, newStatus, patch, ct); + public Task SetProgressAsync(string jobId, int? percent = null, string? message = null, CancellationToken ct = default) => _inner.SetProgressAsync(jobId, percent, message, ct); + public Task IncrementAttemptAsync(string jobId, CancellationToken ct = default) => _inner.IncrementAttemptAsync(jobId, ct); + public Task RequestCancellationAsync(string jobId, CancellationToken ct = default) => _inner.RequestCancellationAsync(jobId, ct); + public Task IsCancellationRequestedAsync(string jobId, CancellationToken ct = default) => _inner.IsCancellationRequestedAsync(jobId, ct); + public Task ScheduleDispatchAsync(ScheduledDispatchState dispatch, CancellationToken ct = default) => _inner.ScheduleDispatchAsync(dispatch, ct); + public Task> ClaimDueDispatchesAsync(DateTimeOffset now, int limit, string nodeId, TimeSpan lease, CancellationToken ct = default) => _inner.ClaimDueDispatchesAsync(now, limit, nodeId, lease, ct); + public Task CompleteDispatchAsync(string dispatchId, string nodeId, CancellationToken ct = default) => _inner.CompleteDispatchAsync(dispatchId, nodeId, ct); + public Task ReleaseDispatchAsync(string dispatchId, string nodeId, DateTimeOffset nextDueUtc, CancellationToken ct = default) => _inner.ReleaseDispatchAsync(dispatchId, nodeId, nextDueUtc, ct); + } +} diff --git a/tests/Foundatio.Tests/Jobs/WorkItemJobTests.cs b/tests/Foundatio.Tests/Jobs/WorkItemJobTests.cs index 9b1cd4226..0e5f6a9ee 100644 --- a/tests/Foundatio.Tests/Jobs/WorkItemJobTests.cs +++ b/tests/Foundatio.Tests/Jobs/WorkItemJobTests.cs @@ -7,8 +7,8 @@ using System.Threading.Tasks; using Exceptionless; using Foundatio.AsyncEx; -using Foundatio.Jobs; -using Foundatio.Messaging; +using Foundatio.Jobs.Legacy; +using Foundatio.Messaging.Legacy; using Foundatio.Queues; using Foundatio.Tests.Extensions; using Foundatio.Xunit; diff --git a/tests/Foundatio.Tests/Locks/InMemoryLockTests.cs b/tests/Foundatio.Tests/Locks/InMemoryLockTests.cs index de9c2d073..24610e3b6 100644 --- a/tests/Foundatio.Tests/Locks/InMemoryLockTests.cs +++ b/tests/Foundatio.Tests/Locks/InMemoryLockTests.cs @@ -2,7 +2,7 @@ using System.Threading.Tasks; using Foundatio.Caching; using Foundatio.Lock; -using Foundatio.Messaging; +using Foundatio.Messaging.Legacy; using Xunit; namespace Foundatio.Tests.Locks; diff --git a/tests/Foundatio.Tests/Messaging/DeliveryIntentTests.cs b/tests/Foundatio.Tests/Messaging/DeliveryIntentTests.cs new file mode 100644 index 000000000..16412f92d --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/DeliveryIntentTests.cs @@ -0,0 +1,197 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.AsyncEx; +using Foundatio.Messaging; +using Foundatio.Tests.Extensions; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class DeliveryIntentTests +{ + [Fact] + public async Task SubscribeAsync_SentOnly_IgnoresPublishedMessagesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + + var received = new ConcurrentQueue(); + var sentSignal = new AsyncCountdownEvent(1); + await using var subscription = await bus.SubscribeAsync((message, _) => + { + received.Enqueue(message.Message.Data); + sentSignal.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Deliveries = MessageDeliveries.Sent }, cts.Token); + + Assert.Equal("", subscription.Source); // no publish channel was wired + Assert.NotEqual("", subscription.Destination); + + // A published event must not reach a sent-only handler (its group does not exist), and the command must. + await bus.PublishAsync(new IntentEvent { Data = "event" }, cancellationToken: cancellationToken); + await bus.SendAsync(new IntentEvent { Data = "command" }, cancellationToken: cancellationToken); + + await sentSignal.WaitAsync(TimeSpan.FromSeconds(2)); + await Task.Delay(TimeSpan.FromMilliseconds(150), cancellationToken); // let any (incorrect) event delivery arrive + + Assert.Equal(["command"], received); + } + + [Fact] + public async Task SubscribeAsync_PublishedOnly_IgnoresSentMessagesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + + var received = new ConcurrentQueue(); + var publishedSignal = new AsyncCountdownEvent(1); + await using var subscription = await bus.SubscribeAsync((message, _) => + { + received.Enqueue(message.Message.Data); + publishedSignal.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Deliveries = MessageDeliveries.Published }, cts.Token); + + Assert.Equal("", subscription.Destination); // no send channel was wired + Assert.NotEqual("", subscription.Source); + + // The command sits unconsumed on its queue (this handler never attached to it); the event must arrive. + await bus.SendAsync(new IntentEvent { Data = "command" }, cancellationToken: cancellationToken); + await bus.PublishAsync(new IntentEvent { Data = "event" }, cancellationToken: cancellationToken); + + await publishedSignal.WaitAsync(TimeSpan.FromSeconds(2)); + await Task.Delay(TimeSpan.FromMilliseconds(150), cancellationToken); // let any (incorrect) command delivery arrive + + Assert.Equal(["event"], received); + + var queueStats = await transport.GetStatsAsync(DestinationAddress.ForQueue("intent-event"), cancellationToken); + Assert.Equal(1, queueStats.Queued); // the command is still parked on the routed queue, untouched + } + + [Fact] + public async Task SubscribeAsync_ExplicitPublished_OnQueueOnlyTransport_ThrowsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var bus = new MessageBus(new QueueOnlyTransport()); + + await Assert.ThrowsAsync(() => bus.SubscribeAsync( + (_, _) => Task.CompletedTask, + new MessageSubscriptionOptions { Deliveries = MessageDeliveries.Published }, + cancellationToken)); + } + + [Fact] + public async Task SubscribeAsync_DefaultBoth_OnQueueOnlyTransport_WiresSendChannelOnlyAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new QueueOnlyTransport(); + await using var bus = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + + var received = new AsyncCountdownEvent(1); + await using var subscription = await bus.SubscribeAsync((message, _) => + { + Assert.Equal("command", message.Message.Data); + received.Signal(); + return Task.CompletedTask; + }, cancellationToken: cts.Token); + + Assert.NotEqual("", subscription.Destination); + Assert.Equal("", subscription.Source); // the publish channel was skipped, not faked + + await bus.SendAsync(new IntentEvent { Data = "command" }, cancellationToken: cancellationToken); + await received.WaitAsync(TimeSpan.FromSeconds(2)); + } + + private sealed class IntentEvent + { + public string? Data { get; set; } + } + + // A transport that truly has no topic/subscription support, so the bus must not wire (or fake) a publish channel. + private sealed class QueueOnlyTransport : IMessageTransport, ISupportsPull, ITransportInfo + { + private readonly ConcurrentDictionary> _queues = new(StringComparer.Ordinal); + private readonly ConcurrentDictionary _inFlight = new(StringComparer.Ordinal); + + public DeliveryGuarantee DeliveryGuarantee => DeliveryGuarantee.AtLeastOnce; + public IReadOnlySet SupportedRoles => new HashSet { DestinationRole.Queue }; + public TransportCapabilities GetCapabilities(DestinationRole role) => TransportCapabilities.None; + + public Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + if (destination.Role != DestinationRole.Queue) + throw new NotSupportedException("Queues only."); + + var queue = _queues.GetOrAdd(destination.Key, static _ => new ConcurrentQueue()); + var items = new List(messages.Count); + foreach (var message in messages) + { + string id = message.MessageId ?? Guid.NewGuid().ToString("N"); + queue.Enqueue(message with { MessageId = id }); + items.Add(new SendItemResult { MessageId = id }); + } + + return Task.FromResult(new SendResult { Items = items }); + } + + public async Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct) + { + var queue = _queues.GetOrAdd(source.Key, static _ => new ConcurrentQueue()); + var deadline = request.MaxWaitTime is { } wait && wait > TimeSpan.Zero ? DateTimeOffset.UtcNow.Add(wait) : DateTimeOffset.UtcNow; + var entries = new List(); + int max = Math.Max(1, request.MaxMessages); + + while (true) + { + while (entries.Count < max && queue.TryDequeue(out var message)) + { + string token = Guid.NewGuid().ToString("N"); + _inFlight[token] = message; + entries.Add(new TransportEntry + { + Id = message.MessageId!, + Destination = source, + Body = message.Body, + Headers = message.Headers, + Receipt = new Receipt { TransportState = token } + }); + } + + if (entries.Count > 0 || DateTimeOffset.UtcNow >= deadline) + return entries; + + await Task.Delay(TimeSpan.FromMilliseconds(15), ct).ConfigureAwait(false); + } + } + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) + { + if (entry.Receipt.TransportState is not string token || !_inFlight.TryRemove(token, out _)) + throw new ReceiptExpiredException(); + + return Task.CompletedTask; + } + + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) + { + if (entry.Receipt.TransportState is not string token || !_inFlight.TryRemove(token, out var message)) + throw new ReceiptExpiredException(); + + _queues.GetOrAdd(entry.Destination.Key, static _ => new ConcurrentQueue()).Enqueue(message); + return Task.CompletedTask; + } + + public ValueTask DisposeAsync() => ValueTask.CompletedTask; + } +} diff --git a/tests/Foundatio.Tests/Messaging/FailureHandlingTests.cs b/tests/Foundatio.Tests/Messaging/FailureHandlingTests.cs new file mode 100644 index 000000000..4ea2857cb --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/FailureHandlingTests.cs @@ -0,0 +1,121 @@ +using System; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class FailureHandlingTests +{ + [Fact] + public async Task DeadLetterOn_MatchingException_DeadLettersOnFirstAttemptAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport); + int attempts = 0; + + await using var subscription = await bus.SubscribeAsync((_, _) => + { + Interlocked.Increment(ref attempts); + throw new ArgumentException("bad data a retry can never fix"); + }, new MessageSubscriptionOptions { MaxAttempts = 5 }.DeadLetterOn(), cancellationToken); + + await bus.SendAsync(new FailingItem { Data = "poison" }, cancellationToken: cancellationToken); + + var stats = await WaitForDeadLetterAsync(transport, "failing-item", cancellationToken); + Assert.Equal(1, stats.Deadletter); + Assert.Equal(1, Volatile.Read(ref attempts)); // never retried + + var dead = Assert.Single(await transport.ReceiveDeadLetteredAsync(DestinationAddress.ForQueue("failing-item"), new ReceiveRequest { MaxMessages = 10 }, cancellationToken)); + Assert.Equal("unrecoverable:ArgumentException", dead.Headers[KnownHeaders.DeadLetterReason]); + } + + [Fact] + public async Task DeadLetterWhen_GlobalPolicy_AppliesWhenSubscriptionDoesNotOverrideAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport, new MessageBusOptions + { + RetryPolicy = new RetryPolicy { DeadLetterWhen = ex => ex is InvalidOperationException } + }); + int attempts = 0; + + await using var subscription = await bus.SubscribeAsync((_, _) => + { + Interlocked.Increment(ref attempts); + throw new InvalidOperationException("unrecoverable per global policy"); + }, cancellationToken: cancellationToken); + + await bus.SendAsync(new FailingItem { Data = "poison" }, cancellationToken: cancellationToken); + + var stats = await WaitForDeadLetterAsync(transport, "failing-item", cancellationToken); + Assert.Equal(1, stats.Deadletter); + Assert.Equal(1, Volatile.Read(ref attempts)); + } + + [Fact] + public async Task DeadLetter_StampsForensicsHeadersAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport); + + await using var subscription = await bus.SubscribeAsync((_, _) => + throw new InvalidOperationException("the failure detail"), + new MessageSubscriptionOptions { MaxAttempts = 1 }, cancellationToken); + + await bus.SendAsync(new FailingItem { Data = "doomed" }, cancellationToken: cancellationToken); + + await WaitForDeadLetterAsync(transport, "failing-item", cancellationToken); + var dead = Assert.Single(await transport.ReceiveDeadLetteredAsync(DestinationAddress.ForQueue("failing-item"), new ReceiveRequest { MaxMessages = 10 }, cancellationToken)); + + Assert.Equal(typeof(InvalidOperationException).FullName, dead.Headers[KnownHeaders.DeadLetterExceptionType]); + Assert.Equal("the failure detail", dead.Headers[KnownHeaders.DeadLetterExceptionMessage]); + Assert.NotEmpty(dead.Headers[KnownHeaders.DeadLetterExceptionStackTrace]); + Assert.Equal("failing-item", dead.Headers[KnownHeaders.DeadLetterOriginalDestination]); + Assert.NotEmpty(dead.Headers[KnownHeaders.DeadLetterFailedAt]); + // The exhausted count is forensics-only: message.attempts is left alone so a replayed message starts fresh. + Assert.Equal("1", dead.Headers[KnownHeaders.DeadLetterAttempts]); + Assert.False(dead.Headers.ContainsKey(KnownHeaders.Attempts)); + } + + [Fact] + public void DefaultBackoff_MatchesTheConvergedCurve() + { + // Immediate first retry, then 10s/20s/30s (capped) with ±20% jitter. + Assert.Equal(TimeSpan.Zero, RetryPolicy.DefaultBackoff(1)); + + foreach ((int attempt, double expectedSeconds) in new[] { (2, 10d), (3, 20d), (4, 30d), (7, 30d) }) + { + var delay = RetryPolicy.DefaultBackoff(attempt); + Assert.InRange(delay.TotalSeconds, expectedSeconds * 0.8, expectedSeconds * 1.2); + } + + // The default policy uses the curve. + Assert.Same(RetryPolicy.DefaultBackoff, new RetryPolicy().Backoff); + } + + private static async Task WaitForDeadLetterAsync(InMemoryMessageTransport transport, string destination, CancellationToken cancellationToken) + { + var address = DestinationAddress.ForQueue(destination); + var stats = await transport.GetStatsAsync(address, cancellationToken); + long deadline = Environment.TickCount64 + 10_000; + while (stats.Deadletter == 0 && Environment.TickCount64 < deadline) + { + await Task.Delay(25, cancellationToken); + stats = await transport.GetStatsAsync(address, cancellationToken); + } + + return stats; + } + + [MessageRoute("failing-item")] + private sealed class FailingItem + { + public string? Data { get; set; } + } +} diff --git a/tests/Foundatio.Tests/Messaging/InMemoryMessageBusTests.cs b/tests/Foundatio.Tests/Messaging/InMemoryMessageBusTests.cs index d94a0a9c8..05d383f78 100644 --- a/tests/Foundatio.Tests/Messaging/InMemoryMessageBusTests.cs +++ b/tests/Foundatio.Tests/Messaging/InMemoryMessageBusTests.cs @@ -1,7 +1,7 @@ using System; using System.Threading.Tasks; using Foundatio.AsyncEx; -using Foundatio.Messaging; +using Foundatio.Messaging.Legacy; using Foundatio.Tests.Extensions; using Xunit; diff --git a/tests/Foundatio.Tests/Messaging/InMemoryMessageTransportTests.cs b/tests/Foundatio.Tests/Messaging/InMemoryMessageTransportTests.cs new file mode 100644 index 000000000..0a4821e37 --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/InMemoryMessageTransportTests.cs @@ -0,0 +1,75 @@ +using System; +using System.Collections.Generic; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class InMemoryMessageTransportTests : MessageTransportConformanceTests +{ + public InMemoryMessageTransportTests(ITestOutputHelper output) : base(output) { } + + protected override IMessageTransport CreateTransport() + { + return new InMemoryMessageTransport(); + } + + [Fact] + public void DestinationAddress_KeyEncodesTopicAndSubscription() + { + var destination = DestinationAddress.ForSubscription("orders", "sub-a"); + Assert.Equal("orders/sub-a", destination.Key); + Assert.Equal("orders", destination.Topic); + Assert.Equal("sub-a", destination.Name); + Assert.Equal(DestinationRole.Subscription, destination.Role); + + // A bare (non-subscription) destination has no topic and a bare key. + var bare = DestinationAddress.ForQueue("orders"); + Assert.Null(bare.Topic); + Assert.Equal("orders", bare.Key); + Assert.NotEqual(destination, bare); + } + + [Fact] + public void MessageHeaders_SerializeToJson_RoundTripsCaseInsensitively() + { + var headers = MessageHeaders.Create([ + new KeyValuePair("Message.Type", "order.created"), + new KeyValuePair("tenant", "acme") + ]); + + // The shared codec both transports use preserves the case-insensitive contract across the wire. + var roundTripped = MessageHeaders.DeserializeFromJson(MessageHeaders.SerializeToJson(headers)); + Assert.Equal("order.created", roundTripped["MESSAGE.TYPE"]); + Assert.Equal("acme", roundTripped["tenant"]); + + Assert.Empty(MessageHeaders.DeserializeFromJson(null)); + Assert.Empty(MessageHeaders.DeserializeFromJson("")); + } + + [Fact] + public void MessageHeaders_AreImmutableAndCaseInsensitive() + { + var source = new Dictionary(StringComparer.Ordinal) + { + ["message.type"] = "order.created" + }; + + var headers = MessageHeaders.Create(source); + source["message.type"] = "changed"; + + Assert.Equal("order.created", headers["MESSAGE.TYPE"]); + Assert.Equal("order.created", headers.GetValueOrDefault("Message.Type")); + Assert.True(headers.ContainsKey("MESSAGE.TYPE")); + + var updated = headers.ToBuilder() + .Set("TraceParent", "00-123") + .SetIfMissing("traceparent", "ignored") + .Build(); + + Assert.Equal("00-123", updated["traceparent"]); + Assert.False(headers.ContainsKey("traceparent")); + } + +} diff --git a/tests/Foundatio.Tests/Messaging/MessageTests.cs b/tests/Foundatio.Tests/Messaging/MessageTests.cs index 67cc60a76..695d9a586 100644 --- a/tests/Foundatio.Tests/Messaging/MessageTests.cs +++ b/tests/Foundatio.Tests/Messaging/MessageTests.cs @@ -1,6 +1,6 @@ using System; using System.Runtime.InteropServices; -using Foundatio.Messaging; +using Foundatio.Messaging.Legacy; using Xunit; namespace Foundatio.Tests.Messaging; diff --git a/tests/Foundatio.Tests/Messaging/MessagingTestHarnessTests.cs b/tests/Foundatio.Tests/Messaging/MessagingTestHarnessTests.cs new file mode 100644 index 000000000..5b73010eb --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/MessagingTestHarnessTests.cs @@ -0,0 +1,173 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; +using Foundatio.Messaging.Testing; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class MessagingTestHarnessTests +{ + [Fact] + public async Task Harness_RecordsSendPublishAndHandledWithTypedAccessAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var harness = new MessagingTestHarness(); + await using var bus = new MessageBus(harness.Transport, new MessageBusOptions { OwnsTransport = false }); + + var handled = new List(); + await using var subscription = await bus.SubscribeAsync((context, _) => + { + lock (handled) + handled.Add(context.Message.Id); + return Task.CompletedTask; + }, cancellationToken: cancellationToken); + + await bus.SendAsync(new HarnessOrder { Id = "cmd" }, cancellationToken: cancellationToken); + await bus.PublishAsync(new HarnessOrder { Id = "evt" }, cancellationToken: cancellationToken); + await harness.WaitForIdleAsync(cancellationToken: cancellationToken); + + // Sends and publishes are recorded separately, deserialized back to the message type. + Assert.Equal("cmd", Assert.Single(harness.Sent()).Id); + Assert.Equal("evt", Assert.Single(harness.Published()).Id); + Assert.Equal(2, harness.Handled().Count); + Assert.Equal(2, handled.Count); + + // Raw recordings carry the route and role for topology assertions. + var sent = Assert.Single(harness.SentMessages); + Assert.Equal("harness-orders", sent.Destination); + Assert.Equal(DestinationRole.Queue, sent.Role); + var published = Assert.Single(harness.PublishedMessages); + Assert.Equal("harness-orders", published.Destination); + Assert.Equal(DestinationRole.Topic, published.Role); + + // Negative assertions are immediate once idle. + Assert.Empty(harness.DeadLetteredMessages); + Assert.Empty(harness.AbandonedMessages); + Assert.Empty(harness.Sent()); + } + + [Fact] + public async Task Harness_RetryCycleEndsInDeadLetterAndIsFullyObservableAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var harness = new MessagingTestHarness(); + await using var bus = new MessageBus(harness.Transport, new MessageBusOptions { OwnsTransport = false }); + + int attempts = 0; + await using var subscription = await bus.SubscribeAsync((_, _) => + { + Interlocked.Increment(ref attempts); + throw new InvalidOperationException("always fails"); + }, new MessageSubscriptionOptions { MaxAttempts = 3, RedeliveryBackoff = _ => TimeSpan.Zero }, cancellationToken); + + await bus.SendAsync(new HarnessOrder { Id = "poison" }, cancellationToken: cancellationToken); + await harness.WaitForIdleAsync(cancellationToken: cancellationToken); + + // The whole failure path is assertable: two retries, then terminal dead-letter with the reason and forensics. + Assert.Equal(3, Volatile.Read(ref attempts)); + Assert.Equal(2, harness.AbandonedMessages.Count); + Assert.All(harness.Abandoned(), m => Assert.Equal("poison", m.Id)); + Assert.Equal(2, harness.Abandoned().Count); + var dead = Assert.Single(harness.DeadLetteredMessages); + Assert.Equal("handler-error", dead.Reason); + Assert.Equal(3, dead.Attempts); + Assert.Equal(typeof(InvalidOperationException).FullName, dead.Headers[KnownHeaders.DeadLetterExceptionType]); + Assert.Equal("poison", Assert.Single(harness.DeadLettered()).Id); + Assert.Empty(harness.HandledMessages); + } + + [Fact] + public async Task WaitForIdle_CoversDelayedRedeliveriesAndTimesOutWithDiagnosticsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var harness = new MessagingTestHarness(); + await using var bus = new MessageBus(harness.Transport, new MessageBusOptions { OwnsTransport = false }); + + int attempts = 0; + await using var subscription = await bus.SubscribeAsync((_, _) => + { + if (Interlocked.Increment(ref attempts) == 1) + throw new InvalidOperationException("fails once"); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { MaxAttempts = 2, RedeliveryBackoff = _ => TimeSpan.FromMilliseconds(500) }, cancellationToken); + + await bus.SendAsync(new HarnessOrder { Id = "retry-me" }, cancellationToken: cancellationToken); + + // The retry is parked in a redelivery timer (neither queued nor in flight); the harness must still see it. + await harness.WaitForIdleAsync(cancellationToken: cancellationToken); + Assert.Equal(2, Volatile.Read(ref attempts)); + Assert.Single(harness.Handled()); + + // Timeout.InfiniteTimeSpan means wait-until-idle; other negative timeouts are rejected up front. + await harness.WaitForIdleAsync(Timeout.InfiniteTimeSpan, cancellationToken); + await Assert.ThrowsAsync(() => harness.WaitForIdleAsync(TimeSpan.FromMilliseconds(-2), cancellationToken)); + + // A destination that never drains fails with the busy destinations named. + await using var stuck = await bus.SubscribeAsync((_, handlerToken) => Task.Delay(Timeout.Infinite, handlerToken), + cancellationToken: cancellationToken); + await bus.SendAsync(new HarnessOther { Id = "stuck" }, cancellationToken: cancellationToken); + + var timeout = await Assert.ThrowsAsync(() => harness.WaitForIdleAsync(TimeSpan.FromSeconds(2), cancellationToken)); + Assert.Contains("harness-other", timeout.Message); + } + + [Fact] + public async Task UseTestHarness_WiresDeclarativeHandlersOverTheRecordingTransportAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + + var services = new ServiceCollection(); + services.AddLogging(); + services.AddFoundatio() + .Messaging.UseTestHarness() + .Messaging.AddHandler(); + + await using var provider = services.BuildServiceProvider(); + var hosted = provider.GetServices().ToList(); + foreach (var service in hosted) + await service.StartAsync(cancellationToken); + + try + { + var harness = provider.GetRequiredService(); + var bus = provider.GetRequiredService(); + Assert.Same(harness.Transport, provider.GetRequiredService()); + + await bus.SendAsync(new HarnessOrder { Id = "from-di" }, cancellationToken: cancellationToken); + await harness.WaitForIdleAsync(cancellationToken: cancellationToken); + + Assert.Equal("from-di", Assert.Single(harness.Sent()).Id); + Assert.Equal("from-di", Assert.Single(harness.Handled()).Id); + Assert.Equal("from-di", Assert.Single(RecordingOrderHandler.Handled)); + } + finally + { + foreach (var service in hosted) + await service.StopAsync(cancellationToken); + } + } + + [MessageRoute("harness-orders")] + public class HarnessOrder { public string Id { get; set; } = ""; } + + [MessageRoute("harness-other")] + public class HarnessOther { public string Id { get; set; } = ""; } + + private sealed class RecordingOrderHandler : IMessageHandler + { + public static readonly List Handled = []; + + public Task HandleAsync(IMessageContext context, CancellationToken cancellationToken) + { + lock (Handled) + Handled.Add(context.Message.Id); + return Task.CompletedTask; + } + } +} diff --git a/tests/Foundatio.Tests/Messaging/PubSubTests.cs b/tests/Foundatio.Tests/Messaging/PubSubTests.cs new file mode 100644 index 000000000..9728a3fef --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/PubSubTests.cs @@ -0,0 +1,483 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.AsyncEx; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Foundatio.Tests.Extensions; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class PubSubTests +{ + [Fact] + public async Task PublishAsync_FansOutToMultipleSubscriptionsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var pubSub = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + + var firstReceived = new AsyncCountdownEvent(1); + var secondReceived = new AsyncCountdownEvent(1); + + await using var first = await pubSub.SubscribeAsync((message, _) => + { + Assert.Equal("published", message.Message.Data); + firstReceived.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "subscriber-a" }, cts.Token); + + await using var second = await pubSub.SubscribeAsync((message, _) => + { + Assert.Equal("published", message.Message.Data); + secondReceived.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "subscriber-b" }, cts.Token); + + await pubSub.PublishAsync(new PreviewEvent { Data = "published" }, cancellationToken: cancellationToken); + + await firstReceived.WaitAsync(TimeSpan.FromSeconds(2)); + await secondReceived.WaitAsync(TimeSpan.FromSeconds(2)); + var firstStats = await transport.GetStatsAsync(DestinationAddress.ForSubscription(first.Topic, first.Subscription), cancellationToken); + var secondStats = await transport.GetStatsAsync(DestinationAddress.ForSubscription(second.Topic, second.Subscription), cancellationToken); + Assert.Equal(1, firstStats.Completed); + Assert.Equal(1, secondStats.Completed); + } + + [Fact] + public async Task SubscribeAsync_WithSameSubscriptionAndDifferentKeys_CompetesOnTransportSubscriptionAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var pubSub = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var received = new AsyncCountdownEvent(2); + var deliveriesByMessageId = new ConcurrentDictionary(StringComparer.Ordinal); + + Func, CancellationToken, Task> handler = (message, _) => + { + deliveriesByMessageId.AddOrUpdate(message.Id, 1, (_, count) => count + 1); + received.Signal(); + return Task.CompletedTask; + }; + + await using var first = await pubSub.SubscribeAsync(handler, new MessageSubscriptionOptions + { + Subscription = "billing-service", + Key = "node-a" + }, cts.Token); + await using var second = await pubSub.SubscribeAsync(handler, new MessageSubscriptionOptions + { + Subscription = "billing-service", + Key = "node-b" + }, cts.Token); + + await pubSub.PublishBatchAsync([ + new PreviewEvent { Data = "one" }, + new PreviewEvent { Data = "two" } + ], cancellationToken: cancellationToken); + + await received.WaitAsync(TimeSpan.FromSeconds(2)); + await WaitForCompletedAsync(transport, DestinationAddress.ForSubscription(first.Topic, first.Subscription), 2, cancellationToken); + + Assert.Equal(first.Topic, second.Topic); + Assert.Equal(first.Subscription, second.Subscription); + Assert.Equal(first.Source, second.Source); // same topic + subscription -> one shared transport source + Assert.NotEqual(first.Key, second.Key); + Assert.Equal(2, deliveriesByMessageId.Count); + Assert.All(deliveriesByMessageId.Values, count => Assert.Equal(1, count)); + } + + [Fact] + public async Task SubscribeAsync_SameSubscriptionOnTwoTopics_IsolatesPerTopicAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var pubSub = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + + var ordersReceived = new List(); + var paymentsReceived = new List(); + var ordersSignal = new AsyncCountdownEvent(1); + var paymentsSignal = new AsyncCountdownEvent(1); + + // The same subscription identity ("shared") on two different topics. + await using var orders = await pubSub.SubscribeAsync((message, _) => + { + lock (ordersReceived) + ordersReceived.Add(message.Message.Data); + ordersSignal.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Topic = "orders", Subscription = "shared" }, cts.Token); + + await using var payments = await pubSub.SubscribeAsync((message, _) => + { + lock (paymentsReceived) + paymentsReceived.Add(message.Message.Data); + paymentsSignal.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Topic = "payments", Subscription = "shared" }, cts.Token); + + Assert.Equal(orders.Subscription, payments.Subscription); // same logical subscription identity + Assert.NotEqual(orders.Source, payments.Source); // but distinct topic-qualified transport sources + + // Publish one message to each topic. Each subscriber must receive only its own topic's message — proving both + // subscribers are live (not an always-broken one passing a negative-only assertion) and that they are isolated. + await pubSub.PublishAsync(new PreviewEvent { Data = "to-orders" }, new MessagePublishOptions { Topic = "orders" }, cancellationToken); + await pubSub.PublishAsync(new PreviewEvent { Data = "to-payments" }, new MessagePublishOptions { Topic = "payments" }, cancellationToken); + + await ordersSignal.WaitAsync(TimeSpan.FromSeconds(2)); + await paymentsSignal.WaitAsync(TimeSpan.FromSeconds(2)); + + // Let any (incorrect) cross-topic delivery arrive before asserting each side received only its own message. + await Task.Delay(TimeSpan.FromMilliseconds(150), cancellationToken); + + Assert.Equal(new[] { "to-orders" }, ordersReceived); + Assert.Equal(new[] { "to-payments" }, paymentsReceived); + } + + [Fact] + public async Task PublishBatchAsync_DeliversAllMessagesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var pubSub = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var received = new AsyncCountdownEvent(2); + + await using var subscription = await pubSub.SubscribeAsync((message, _) => + { + Assert.StartsWith("batch-", message.Message.Data); + received.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "batch-subscription" }, cts.Token); + + await pubSub.PublishBatchAsync([ + new PreviewEvent { Data = "batch-one" }, + new PreviewEvent { Data = "batch-two" } + ], cancellationToken: cancellationToken); + + await received.WaitAsync(TimeSpan.FromSeconds(2)); + var stats = await transport.GetStatsAsync(DestinationAddress.ForSubscription(subscription.Topic, subscription.Subscription), cancellationToken); + Assert.Equal(2, stats.Completed); + } + + [Fact] + public async Task PublishAsync_WithOptions_PropagatesHeadersAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var pubSub = new MessageBus(new InMemoryMessageTransport()); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var received = new TaskCompletionSource>(TaskCreationOptions.RunContinuationsAsynchronously); + + await using var subscription = await pubSub.SubscribeAsync((message, _) => + { + received.TrySetResult(message); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "metadata-subscription" }, cts.Token); + + await pubSub.PublishAsync(new PreviewEvent { Data = "metadata" }, new MessagePublishOptions + { + CorrelationId = "corr-456", + Priority = MessagePriority.High, + Headers = MessageHeaders.Create([ + new KeyValuePair("tenant", "acme") + ]) + }, cancellationToken); + + var completed = await Task.WhenAny(received.Task, Task.Delay(TimeSpan.FromSeconds(2), cancellationToken)); + Assert.Equal(received.Task, completed); + + var message = await received.Task; + Assert.Equal("metadata", message.Message.Data); + Assert.Equal("corr-456", message.CorrelationId); + Assert.Equal(MessagePriority.High, message.Priority); + Assert.Equal("acme", message.Headers["tenant"]); + Assert.Equal(typeof(PreviewEvent).FullName, message.MessageType); + + } + + [Fact] + public async Task PublishAsync_WithDelay_SchedulesThroughRuntimeStoreAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + await using var transport = new InMemoryMessageTransport(); + await using var pubSub = new MessageBus(transport, new MessageBusOptions { RuntimeStore = store }); + var processor = CreateDispatchProcessor(store, transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var received = new AsyncCountdownEvent(1); + + await using var subscription = await pubSub.SubscribeAsync((_, _) => + { + received.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "delayed-subscription" }, cts.Token); + + await pubSub.PublishAsync(new PreviewEvent { Data = "later" }, new MessagePublishOptions { Delay = TimeSpan.FromMinutes(1) }, cancellationToken); + + await Assert.ThrowsAsync(async () => await received.WaitAsync(TimeSpan.FromMilliseconds(50))); + Assert.Equal(1, await processor.RunDueOccurrencesAsync(DateTimeOffset.UtcNow.AddMinutes(2), cancellationToken: cancellationToken)); + await received.WaitAsync(TimeSpan.FromSeconds(2)); + + } + + [Fact] + public async Task SubscribeAsync_WhenHandlerFails_RedeliversAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var pubSub = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var received = new AsyncCountdownEvent(2); + int attempts = 0; + + await using var subscription = await pubSub.SubscribeAsync((message, _) => + { + attempts++; + Assert.Equal(attempts, message.Attempts); + received.Signal(); + + if (attempts == 1) + throw new InvalidOperationException("try again"); + + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "retry-subscription", MaxAttempts = 2 }, cts.Token); + + await pubSub.PublishAsync(new PreviewEvent { Data = "retry" }, cancellationToken: cancellationToken); + + await received.WaitAsync(TimeSpan.FromSeconds(2)); + var stats = await transport.GetStatsAsync(DestinationAddress.ForSubscription(subscription.Topic, subscription.Subscription), cancellationToken); + Assert.Equal(1, stats.Completed); + Assert.Equal(1, stats.Abandoned); + } + + + [Fact] + public async Task SubscribeAsync_WithSameKeyAndSameRegistration_SharesTheUnderlyingConsumerAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var pubSub = new MessageBus(new InMemoryMessageTransport()); + int handled = 0; + var received = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + Func, CancellationToken, Task> handler = (_, _) => + { + Interlocked.Increment(ref handled); + received.TrySetResult(); + return Task.CompletedTask; + }; + + // Registering the same key + handler + options twice is idempotent: both handles refer to the one underlying + // consumer, so a published message is handled exactly once. + await using var first = await pubSub.SubscribeAsync(handler, new MessageSubscriptionOptions { Subscription = "same-key", Key = "shared" }, cancellationToken); + await using var second = await pubSub.SubscribeAsync(handler, new MessageSubscriptionOptions { Subscription = "same-key", Key = "shared" }, cancellationToken); + + Assert.Equal(first.Key, second.Key); + Assert.Equal(first.Source, second.Source); + + await pubSub.PublishAsync(new PreviewEvent { Data = "once" }, cancellationToken: cancellationToken); + await received.Task.WaitAsync(TimeSpan.FromSeconds(10), cancellationToken); + await Task.Delay(250, cancellationToken); + Assert.Equal(1, Volatile.Read(ref handled)); + } + + [Fact] + public async Task SubscribeAsync_WithSameKeyAndDifferentHandler_ThrowsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var pubSub = new MessageBus(new InMemoryMessageTransport()); + + await using var first = await pubSub.SubscribeAsync((_, _) => Task.CompletedTask, new MessageSubscriptionOptions { Subscription = "same-key", Key = "shared" }, cancellationToken); + + await Assert.ThrowsAsync(async () => + await pubSub.SubscribeAsync((_, _) => Task.CompletedTask, new MessageSubscriptionOptions { Subscription = "same-key", Key = "shared" }, cancellationToken)); + } + + [Fact] + public async Task SubscribeAsync_WithSameKeyAndDifferentFailurePolicy_ThrowsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var pubSub = new MessageBus(new InMemoryMessageTransport()); + Func, CancellationToken, Task> handler = (_, _) => Task.CompletedTask; + + await using var first = await pubSub.SubscribeAsync(handler, new MessageSubscriptionOptions + { + Subscription = "same-key", + Key = "shared", + DeadLetterWhen = static ex => ex is InvalidOperationException + }, cancellationToken); + + // Shared-key subscriptions form ONE competing group; members with different retry/dead-letter LOGIC would + // settle the same message differently depending on who received it, so a divergent policy must be rejected — + // by delegate identity, not by mere has-a-policy presence. + await Assert.ThrowsAsync(async () => + await pubSub.SubscribeAsync(handler, new MessageSubscriptionOptions + { + Subscription = "same-key", + Key = "shared", + DeadLetterWhen = static ex => ex is ArgumentException + }, cancellationToken)); + } + + [Fact] + public async Task SubscribeAsync_WithGroupedTopicAndSubscriptionIdentity_ReceivesRawMessagesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + var routing = new MessageRoutingOptionsBuilder() + .MapTopic("order-events", typeof(IGroupedEvent)) + .UseSubscriptionIdentity("billing-service") + .Build(); + await using var pubSub = new MessageBus(transport, new MessageBusOptions { Router = new DefaultMessageRouter(routing) }); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var received = new AsyncCountdownEvent(2); + var messageTypes = new List(); + + await using var subscription = await pubSub.SubscribeAsync((message, _) => + { + lock (messageTypes) + messageTypes.Add(message.MessageType!); + + received.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { RouteType = typeof(IGroupedEvent) }, cts.Token); + + await pubSub.PublishBatchAsync(new object[] + { + new PreviewEvent { Data = "one" }, + new OtherEvent { Data = "two" } + }, cancellationToken: cancellationToken); + + await received.WaitAsync(TimeSpan.FromSeconds(2)); + + Assert.Equal("order-events", subscription.Topic); + Assert.Equal("billing-service", subscription.Subscription); + Assert.Equal("order-events/billing-service", subscription.Source); // topic-qualified transport source + Assert.Contains(typeof(PreviewEvent).FullName!, messageTypes); + Assert.Contains(typeof(OtherEvent).FullName!, messageTypes); + + var stats = await transport.GetStatsAsync(DestinationAddress.ForSubscription(subscription.Topic, subscription.Subscription), cancellationToken); + Assert.Equal(2, stats.Completed); + } + + + [Fact] + public async Task PublishAsync_WithDelay_OnTopicWithoutNativeDelay_RoutesThroughRuntimeStoreAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + + // The AWS SQS/SNS shape: queues honor a native delay (15-minute cap) but topics have none. A delayed publish + // within the QUEUE ceiling must still route through the runtime store — deciding by transport-wide capability + // would take the native path and the broker would silently drop the delay. + var store = new InMemoryJobRuntimeStore(); + await using var transport = new RoleSplitDelayTransport(queueMaxDelay: TimeSpan.FromMinutes(15)); + await using var pubSub = new MessageBus(transport, new MessageBusOptions { RuntimeStore = store }); + var processor = CreateDispatchProcessor(store, transport); + + await pubSub.PublishAsync(new PreviewEvent { Data = "later" }, new MessagePublishOptions { Delay = TimeSpan.FromMinutes(5) }, cancellationToken); + + Assert.Equal(0, transport.SendCount); + Assert.Equal(1, await processor.RunDueOccurrencesAsync(DateTimeOffset.UtcNow.AddMinutes(10), cancellationToken: cancellationToken)); + Assert.Equal(1, transport.SendCount); + Assert.Equal(DestinationRole.Topic, transport.LastDestination?.Role); + Assert.Null(transport.LastSendOptions?.DeliverAt); // the store dispatches it as due; the delay is spent, not forwarded + + // A delayed QUEUE send within the same transport's queue ceiling still uses the native path. + await pubSub.SendAsync(new PreviewEvent { Data = "soon" }, new MessageSendOptions { Delay = TimeSpan.FromMinutes(5) }, cancellationToken); + Assert.Equal(2, transport.SendCount); + Assert.NotNull(transport.LastSendOptions?.DeliverAt); + } + + private static async Task WaitForCompletedAsync(InMemoryMessageTransport transport, DestinationAddress destination, long expected, CancellationToken cancellationToken) + { + var deadline = DateTimeOffset.UtcNow.AddSeconds(2); + while (DateTimeOffset.UtcNow < deadline) + { + var stats = await transport.GetStatsAsync(destination, cancellationToken); + if (stats.Completed == expected) + return; + + await Task.Delay(TimeSpan.FromMilliseconds(10), cancellationToken); + } + + var finalStats = await transport.GetStatsAsync(destination, cancellationToken); + Assert.Equal(expected, finalStats.Completed); + } + + private static JobScheduleProcessor CreateDispatchProcessor(IJobRuntimeStore store, IMessageTransport transport) + { + var serviceProvider = new ServiceCollection().BuildServiceProvider(); + var worker = new JobWorker(store, serviceProvider, nodeId: "node-a"); + return new JobScheduleProcessor(new InMemoryJobScheduler(), store, worker, nodeId: "node-a", transport: transport); + } + + // Mirrors AWS SQS/SNS: native delayed delivery on queues only. Topic sends with a future DeliverAt throw, so a + // silent delay drop cannot hide. + private sealed class RoleSplitDelayTransport : IMessageTransport, ISupportsPull, ITransportInfo + { + private readonly TimeSpan _queueMaxDelay; + + public RoleSplitDelayTransport(TimeSpan queueMaxDelay) => _queueMaxDelay = queueMaxDelay; + + public int SendCount { get; private set; } + public TransportSendOptions? LastSendOptions { get; private set; } + public DestinationAddress? LastDestination { get; private set; } + + public DeliveryGuarantee DeliveryGuarantee => DeliveryGuarantee.AtLeastOnce; + public IReadOnlySet SupportedRoles => + new HashSet { DestinationRole.Queue, DestinationRole.Topic, DestinationRole.Subscription }; + + public TransportCapabilities GetCapabilities(DestinationRole role) => role == DestinationRole.Topic + ? TransportCapabilities.None + : new TransportCapabilities { DelayedDelivery = true, MaxDeliveryDelay = _queueMaxDelay }; + + public Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + if (destination.Role == DestinationRole.Topic && options.DeliverAt is { } deliverAt && deliverAt > DateTimeOffset.UtcNow) + throw new NotSupportedException("Topics have no native delayed delivery."); + + SendCount += messages.Count; + LastSendOptions = options; + LastDestination = destination; + var items = new SendItemResult[messages.Count]; + for (int i = 0; i < messages.Count; i++) + items[i] = new SendItemResult { MessageId = messages[i].MessageId ?? Guid.NewGuid().ToString("N") }; + + return Task.FromResult(new SendResult { Items = items }); + } + + public Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct) + => Task.FromResult>([]); + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + public ValueTask DisposeAsync() => ValueTask.CompletedTask; + } + + private interface IGroupedEvent + { + } + + private sealed class PreviewEvent : IGroupedEvent + { + public string? Data { get; set; } + } + + private sealed class OtherEvent : IGroupedEvent + { + public string? Data { get; set; } + } +} diff --git a/tests/Foundatio.Tests/Messaging/TopologyModeTests.cs b/tests/Foundatio.Tests/Messaging/TopologyModeTests.cs new file mode 100644 index 000000000..d9d2752d0 --- /dev/null +++ b/tests/Foundatio.Tests/Messaging/TopologyModeTests.cs @@ -0,0 +1,93 @@ +using System; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.AsyncEx; +using Foundatio.Messaging; +using Foundatio.Tests.Extensions; +using Xunit; + +namespace Foundatio.Tests.Messaging; + +public class TopologyModeTests +{ + [Fact] + public async Task Validate_WithPreProvisionedTopology_DeliversAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + + // Provision out of band (the admin path), then run the bus in validate-only mode. + var topic = DestinationAddress.ForTopic("topology-event"); + var subscription = DestinationAddress.ForSubscription("topology-event", "svc"); + await transport.EnsureAsync([new DestinationDeclaration { Address = topic }, new DestinationDeclaration { Address = subscription }], cancellationToken); + + await using var bus = new MessageBus(transport, new MessageBusOptions { Topology = TopologyMode.Validate, OwnsTransport = false }); + var received = new AsyncCountdownEvent(1); + await using var handle = await bus.SubscribeAsync((message, _) => + { + Assert.Equal("hello", message.Message.Data); + received.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "svc" }, cts.Token); + + await bus.PublishAsync(new TopologyEvent { Data = "hello" }, cancellationToken: cancellationToken); + await received.WaitAsync(TimeSpan.FromSeconds(2)); + } + + [Fact] + public async Task Validate_WithMissingTopology_ThrowsAndCreatesNothingAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport, new MessageBusOptions { Topology = TopologyMode.Validate, OwnsTransport = false }); + + await Assert.ThrowsAsync(() => bus.PublishAsync(new TopologyEvent { Data = "hello" }, cancellationToken: cancellationToken)); + await Assert.ThrowsAsync(() => bus.SubscribeAsync((_, _) => Task.CompletedTask, cancellationToken: cancellationToken)); + + Assert.False(await transport.ExistsAsync(DestinationAddress.ForTopic("topology-event"), cancellationToken)); + } + + [Fact] + public async Task None_NeverTouchesTopologyAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var bus = new MessageBus(transport, new MessageBusOptions { Topology = TopologyMode.None, OwnsTransport = false }); + + // Publishing to a topic that was never provisioned must not create it (real pub/sub drop semantics). + await bus.PublishAsync(new TopologyEvent { Data = "dropped" }, cancellationToken: cancellationToken); + Assert.False(await transport.ExistsAsync(DestinationAddress.ForTopic("topology-event"), cancellationToken)); + } + + [Fact] + public async Task None_WithPreProvisionedTopology_DeliversAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + + var topic = DestinationAddress.ForTopic("topology-event"); + var subscription = DestinationAddress.ForSubscription("topology-event", "svc"); + await transport.EnsureAsync([new DestinationDeclaration { Address = topic }, new DestinationDeclaration { Address = subscription }], cancellationToken); + + await using var bus = new MessageBus(transport, new MessageBusOptions { Topology = TopologyMode.None, OwnsTransport = false }); + var received = new AsyncCountdownEvent(1); + await using var handle = await bus.SubscribeAsync((message, _) => + { + received.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { Subscription = "svc" }, cts.Token); + + await bus.PublishAsync(new TopologyEvent { Data = "hello" }, cancellationToken: cancellationToken); + await received.WaitAsync(TimeSpan.FromSeconds(2)); + } + + [MessageRoute("topology-event")] + private sealed class TopologyEvent + { + public string? Data { get; set; } + } +} diff --git a/tests/Foundatio.Tests/Queue/BasicQueueTransport.cs b/tests/Foundatio.Tests/Queue/BasicQueueTransport.cs new file mode 100644 index 000000000..af0f7a4a8 --- /dev/null +++ b/tests/Foundatio.Tests/Queue/BasicQueueTransport.cs @@ -0,0 +1,170 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using Foundatio.Messaging; + +namespace Foundatio.Tests.Queue; + +/// +/// A deliberately minimal transport for tests: basic competing-consumer pull semantics only — no native redelivery +/// delay, lock renewal, visibility timeout, delayed delivery, priority, or expiration. Headers are treated as opaque +/// (preserved but never interpreted) and the delivery count is owned solely by the transport, so it never seeds the +/// count from the message.attempts header. This models a real provider that lacks time-based capabilities, +/// which exercises the runtime-store fallbacks and proves the core reconciles the attempt count itself rather than +/// relying on the transport to honor the header. +/// +internal sealed class BasicQueueTransport : IMessageTransport, ISupportsPull, ISupportsDeadLetter, ISupportsStats +{ + private readonly ConcurrentDictionary _destinations = new(StringComparer.OrdinalIgnoreCase); + + public Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + var dest = _destinations.GetOrAdd(destination.Key, static _ => new Destination()); + var results = new SendItemResult[messages.Count]; + for (int index = 0; index < messages.Count; index++) + { + var message = messages[index]; + string id = message.MessageId ?? Guid.NewGuid().ToString("N"); + dest.Ready.Enqueue(new StoredEntry(id, message.Body, message.Headers, DeliveryCount: 1)); + Interlocked.Increment(ref dest.Enqueued); + results[index] = new SendItemResult { MessageId = id }; + } + + return Task.FromResult(new SendResult { Items = results }); + } + + public async Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct) + { + var dest = _destinations.GetOrAdd(source.Key, static _ => new Destination()); + int max = request.MaxMessages <= 0 ? 1 : request.MaxMessages; + DateTimeOffset? deadline = request.MaxWaitTime is { } wait && wait > TimeSpan.Zero ? DateTimeOffset.UtcNow.Add(wait) : null; + var entries = new List(max); + + while (true) + { + while (entries.Count < max && dest.Ready.TryDequeue(out var stored)) + { + string token = Guid.NewGuid().ToString("N"); + dest.InFlight[token] = stored; + Interlocked.Increment(ref dest.Dequeued); + entries.Add(new TransportEntry + { + Id = stored.Id, + Destination = source, + Body = stored.Body, + Headers = stored.Headers, + DeliveryCount = stored.DeliveryCount, + Receipt = new Receipt { TransportState = new BasicReceipt(source.Key, token) } + }); + } + + if (entries.Count > 0 || deadline is null || DateTimeOffset.UtcNow >= deadline) + return entries; + + await Task.Delay(TimeSpan.FromMilliseconds(15), ct).ConfigureAwait(false); + } + } + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) + { + var (dest, token) = Locate(entry); + if (!dest.InFlight.TryRemove(token, out _)) + throw new ReceiptExpiredException(); + + Interlocked.Increment(ref dest.Completed); + return Task.CompletedTask; + } + + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) + { + var (dest, token) = Locate(entry); + if (!dest.InFlight.TryRemove(token, out var stored)) + throw new ReceiptExpiredException(); + + Interlocked.Increment(ref dest.Abandoned); + dest.Ready.Enqueue(stored with { DeliveryCount = stored.DeliveryCount + 1 }); + return Task.CompletedTask; + } + + public Task DeadLetterAsync(TransportEntry entry, string? reason, CancellationToken ct) + { + var (dest, token) = Locate(entry); + if (!dest.InFlight.TryRemove(token, out var stored)) + throw new ReceiptExpiredException(); + + var headers = String.IsNullOrEmpty(reason) ? stored.Headers : stored.Headers.ToBuilder().Set(KnownHeaders.DeadLetterReason, reason).Build(); + dest.Dead.Enqueue(stored with { Headers = headers }); + return Task.CompletedTask; + } + + public Task> ReceiveDeadLetteredAsync(DestinationAddress destination, ReceiveRequest request, CancellationToken ct) + { + var entries = new List(); + if (_destinations.TryGetValue(destination.Key, out var dest)) + { + int max = request.MaxMessages <= 0 ? 1 : request.MaxMessages; + while (entries.Count < max && dest.Dead.TryDequeue(out var stored)) + { + entries.Add(new TransportEntry + { + Id = stored.Id, + Destination = destination, + Body = stored.Body, + Headers = stored.Headers, + DeliveryCount = stored.DeliveryCount, + Receipt = new Receipt { TransportState = null } + }); + } + } + + return Task.FromResult>(entries); + } + + public Task GetStatsAsync(DestinationAddress destination, CancellationToken ct) + { + if (!_destinations.TryGetValue(destination.Key, out var dest)) + return Task.FromResult(new MessageDestinationStats()); + + return Task.FromResult(new MessageDestinationStats + { + Queued = dest.Ready.Count, + Working = dest.InFlight.Count, + Deadletter = dest.Dead.Count, + Enqueued = Interlocked.Read(ref dest.Enqueued), + Dequeued = Interlocked.Read(ref dest.Dequeued), + Completed = Interlocked.Read(ref dest.Completed), + Abandoned = Interlocked.Read(ref dest.Abandoned) + }); + } + + public ValueTask DisposeAsync() + { + _destinations.Clear(); + return ValueTask.CompletedTask; + } + + private (Destination Destination, string Token) Locate(TransportEntry entry) + { + if (entry.Receipt.TransportState is not BasicReceipt receipt || !_destinations.TryGetValue(receipt.Destination, out var dest)) + throw new ReceiptExpiredException(); + + return (dest, receipt.Token); + } + + private sealed record StoredEntry(string Id, ReadOnlyMemory Body, MessageHeaders Headers, int DeliveryCount); + + private sealed record BasicReceipt(string Destination, string Token); + + private sealed class Destination + { + public readonly ConcurrentQueue Ready = new(); + public readonly ConcurrentQueue Dead = new(); + public readonly ConcurrentDictionary InFlight = new(StringComparer.Ordinal); + public long Enqueued; + public long Dequeued; + public long Completed; + public long Abandoned; + } +} diff --git a/tests/Foundatio.Tests/Queue/MessageQueueTests.cs b/tests/Foundatio.Tests/Queue/MessageQueueTests.cs new file mode 100644 index 000000000..aa6213ec0 --- /dev/null +++ b/tests/Foundatio.Tests/Queue/MessageQueueTests.cs @@ -0,0 +1,1109 @@ +using System; +using System.Collections.Concurrent; +using System.Collections.Generic; +using System.Threading.Channels; +using System.Threading; +using System.Threading.Tasks; +using Foundatio; +using Foundatio.AsyncEx; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Foundatio.Tests.Extensions; +using Microsoft.Extensions.DependencyInjection; +using Xunit; + +namespace Foundatio.Tests.Queue; + +public class MessageQueueTests +{ + [Fact] + public async Task EnqueueAsync_WithOptions_CanReceiveAndCompleteAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport); + + string id = await queue.SendAsync(new PreviewWorkItem { Data = "hello" }, new MessageSendOptions + { + CorrelationId = "corr-123", + Priority = MessagePriority.High, + Headers = MessageHeaders.Create([ + new KeyValuePair("tenant", "acme") + ]) + }, cancellationToken); + + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + var received = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + + Assert.NotNull(received); + Assert.Equal(id, received.Id); + Assert.Equal("hello", received.Message.Data); + Assert.Equal("corr-123", received.CorrelationId); + Assert.Equal(MessagePriority.High, received.Priority); + Assert.Equal(1, received.Attempts); + Assert.Equal("acme", received.Headers["tenant"]); + Assert.Equal(typeof(PreviewWorkItem).FullName, received.MessageType); + + await received.CompleteAsync(cancellationToken); + + var stats = await transport.GetStatsAsync(DestinationAddress.ForQueue("preview-work-item"), cancellationToken); + Assert.Equal(1, stats.Completed); + Assert.Equal(0, stats.Working); + } + + [Fact] + public async Task EnqueueBatchAsync_UsesDestinationOverrideAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport); + + await queue.SendBatchAsync([ + new PreviewWorkItem { Data = "one" }, + new PreviewWorkItem { Data = "two" } + ], new MessageSendOptions { Destination = "custom-work" }, cancellationToken); + + await using var collector = await MessageCollector.StartAsync(queue, destination: "custom-work", cancellationToken: cancellationToken); + var first = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + var second = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + + Assert.NotNull(first); + Assert.NotNull(second); + Assert.Equal("one", first.Message.Data); + Assert.Equal("two", second.Message.Data); + + await first.CompleteAsync(cancellationToken); + await second.CompleteAsync(cancellationToken); + } + + [Fact] + public async Task RejectAsync_NonTerminal_RedeliversAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var queue = new MessageBus(new InMemoryMessageTransport()); + await queue.SendAsync(new PreviewWorkItem { Data = "retry" }, cancellationToken: cancellationToken); + + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + var first = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(first); + + await first.RejectAsync(cancellationToken: cancellationToken); + + var second = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(second); + Assert.Equal(first.Id, second.Id); + Assert.Equal(2, second.Attempts); + Assert.Equal("retry", second.Message.Data); + + await second.CompleteAsync(cancellationToken); + } + + [Fact] + public async Task RenewLockAsync_WhenUnsupported_ThrowsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + // BasicQueueTransport intentionally does not implement ISupportsLockRenewal, so the core must surface the + // unsupported capability rather than silently no-op. + await using var queue = new MessageBus(new BasicQueueTransport()); + + await queue.SendAsync(new PreviewWorkItem { Data = "lock" }, cancellationToken: cancellationToken); + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + var message = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(message); + + await Assert.ThrowsAsync(async () => await message.RenewLockAsync(cancellationToken: cancellationToken)); + } + + [Fact] + public async Task RejectAsync_Terminal_DeadLettersAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport); + + await queue.SendAsync(new PreviewWorkItem { Data = "bad" }, cancellationToken: cancellationToken); + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + var message = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(message); + + await message.RejectAsync(new RejectOptions { Terminal = true, Reason = "validation" }, cancellationToken); + + var stats = await transport.GetStatsAsync(DestinationAddress.ForQueue("preview-work-item"), cancellationToken); + Assert.Equal(1, stats.Deadletter); + Assert.Equal(0, stats.Working); + } + + [Fact] + public async Task StartConsumerAsync_WithAutoAck_CompletesMessageAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var handled = new AsyncCountdownEvent(1); + + await using var consumer = await queue.SubscribeAsync((message, _) => + { + Assert.Equal("work", message.Message.Data); + handled.Signal(); + return Task.CompletedTask; + }, cancellationToken: cts.Token); + + await queue.SendAsync(new PreviewWorkItem { Data = "work" }, cancellationToken: cts.Token); + await handled.WaitAsync(TimeSpan.FromSeconds(2)); + await WaitForCompletedAsync(transport, "preview-work-item", cancellationToken); + } + + [Fact] + public async Task StartConsumerAsync_WithManualAck_DoesNotAutoCompleteAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var handled = new AsyncCountdownEvent(1); + + await using var consumer = await queue.SubscribeAsync((message, _) => + { + handled.Signal(); + return Task.CompletedTask; // intentionally does NOT settle the message + }, new MessageSubscriptionOptions { AckMode = AckMode.Manual }, cts.Token); + + await queue.SendAsync(new PreviewWorkItem { Data = "manual" }, cancellationToken: cts.Token); + await handled.WaitAsync(TimeSpan.FromSeconds(2)); + await Task.Delay(200, cts.Token); + + // Manual ack: the handler ran but did not settle, so the message stays in flight and is not auto-completed. + var stats = await transport.GetStatsAsync(DestinationAddress.ForQueue("preview-work-item"), cancellationToken); + Assert.Equal(0, stats.Completed); + Assert.Equal(1, stats.Working); + } + + [Fact] + public async Task StartConsumerAsync_WithPoisonMessage_DeadLettersAndKeepsConsumingAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var handled = new AsyncCountdownEvent(1); + + await using var consumer = await queue.SubscribeAsync((message, _) => + { + Assert.Equal("good", message.Message.Data); + handled.Signal(); + return Task.CompletedTask; + }, cancellationToken: cts.Token); + + // A poison (undeserializable) payload must be dead-lettered without tearing down the consumer loop, so the + // subsequent valid message is still delivered. + await transport.SendAsync(DestinationAddress.ForQueue("preview-work-item"), [ + new TransportMessage { Body = System.Text.Encoding.UTF8.GetBytes("}{ not json"), Headers = MessageHeaders.Empty } + ], new TransportSendOptions(), cts.Token); + await queue.SendAsync(new PreviewWorkItem { Data = "good" }, cancellationToken: cts.Token); + + await handled.WaitAsync(TimeSpan.FromSeconds(5)); + Assert.Equal(0, handled.CurrentCount); + } + + [Fact] + public async Task EnqueueBatchAsync_RespectsTransportMaxBatchSizeAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var transport = new BatchLimitTransport(maxBatchSize: 2); + await using var queue = new MessageBus(transport); + + await queue.SendBatchAsync(new[] + { + new PreviewWorkItem { Data = "1" }, + new PreviewWorkItem { Data = "2" }, + new PreviewWorkItem { Data = "3" }, + new PreviewWorkItem { Data = "4" }, + new PreviewWorkItem { Data = "5" } + }, cancellationToken: cancellationToken); + + // Five messages to one destination with MaxBatchSize=2 must be split into chunks of 2, 2, 1. + Assert.Equal(new[] { 2, 2, 1 }, transport.SendBatchSizes); + } + + [Fact] + public async Task EnqueueAsync_WithDelay_SchedulesThroughRuntimeStoreAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport, new MessageBusOptions { RuntimeStore = store }); + var processor = CreateDispatchProcessor(store, transport); + + await queue.SendAsync(new PreviewWorkItem { Data = "later" }, new MessageSendOptions { Delay = TimeSpan.FromMinutes(1) }, cancellationToken); + + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + var immediate = await collector.NextAsync(TimeSpan.FromMilliseconds(250), cancellationToken); + Assert.Null(immediate); // parked in the runtime store, not on the transport + + Assert.Equal(1, await processor.RunDueOccurrencesAsync(DateTimeOffset.UtcNow.AddMinutes(2), cancellationToken: cancellationToken)); + + var delayed = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(delayed); + Assert.Equal("later", delayed.Message.Data); + await delayed.CompleteAsync(cancellationToken); + } + + [Fact] + public async Task EnqueueAsync_WithDelayAndNoRuntimeStore_ThrowsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var queue = new MessageBus(new InMemoryMessageTransport()); + + await Assert.ThrowsAsync(async () => + await queue.SendAsync(new PreviewWorkItem { Data = "later" }, new MessageSendOptions { Delay = TimeSpan.FromMinutes(1) }, cancellationToken)); + } + + [Fact] + public async Task EnqueueAsync_WithDelay_RespectsTransportMaxDeliveryDelayAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + + // Within the transport's advertised maximum: delivered natively, never touches the runtime store. + var nativeStore = new InMemoryJobRuntimeStore(); + await using var nativeTransport = new CappedDelayTransport(maxDeliveryDelay: TimeSpan.FromMinutes(15)); + await using var nativeQueue = new MessageBus(nativeTransport, new MessageBusOptions { RuntimeStore = nativeStore }); + var nativeProcessor = CreateDispatchProcessor(nativeStore, nativeTransport); + + await nativeQueue.SendAsync(new PreviewWorkItem { Data = "soon" }, new MessageSendOptions { Delay = TimeSpan.FromMinutes(5) }, cancellationToken); + + Assert.Equal(1, nativeTransport.SendCount); + Assert.NotNull(nativeTransport.LastSendOptions?.DeliverAt); + Assert.Equal(0, await nativeProcessor.RunDueOccurrencesAsync(DateTimeOffset.UtcNow.AddYears(1), cancellationToken: cancellationToken)); + + // Beyond the transport's maximum: routed through the runtime store instead of being silently truncated. + var fallbackStore = new InMemoryJobRuntimeStore(); + await using var fallbackTransport = new CappedDelayTransport(maxDeliveryDelay: TimeSpan.FromMinutes(15)); + await using var fallbackQueue = new MessageBus(fallbackTransport, new MessageBusOptions { RuntimeStore = fallbackStore }); + var fallbackProcessor = CreateDispatchProcessor(fallbackStore, fallbackTransport); + + await fallbackQueue.SendAsync(new PreviewWorkItem { Data = "later" }, new MessageSendOptions { Delay = TimeSpan.FromHours(1) }, cancellationToken); + + Assert.Equal(0, fallbackTransport.SendCount); + Assert.Equal(1, await fallbackProcessor.RunDueOccurrencesAsync(DateTimeOffset.UtcNow.AddHours(2), cancellationToken: cancellationToken)); + Assert.Equal(1, fallbackTransport.SendCount); + + await using var collector = await MessageCollector.StartAsync(fallbackQueue, cancellationToken: cancellationToken); + var delayed = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(delayed); + Assert.Equal("later", delayed.Message.Data); + await delayed.CompleteAsync(cancellationToken); + } + + [Fact] + public async Task StartConsumerAsync_WithRedeliveryBackoff_SchedulesRetryThroughRuntimeStoreAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + // BasicQueueTransport lacks native redelivery delay, so the backoff routes through the runtime store. It also + // never seeds the delivery count from the message.attempts header, proving the core reconciles the attempt + // count from the header itself (second attempt must observe Attempts == 2, not a reset-to-1 loop). + await using var transport = new BasicQueueTransport(); + await using var queue = new MessageBus(transport, new MessageBusOptions { RuntimeStore = store }); + var processor = CreateDispatchProcessor(store, transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + var firstAttempt = new AsyncCountdownEvent(1); + var secondAttempt = new AsyncCountdownEvent(1); + int attempts = 0; + + await using var consumer = await queue.SubscribeAsync((message, _) => + { + attempts++; + if (attempts == 1) + { + Assert.Equal(1, message.Attempts); + firstAttempt.Signal(); + throw new InvalidOperationException("try again later"); + } + + Assert.Equal(2, message.Attempts); + Assert.Equal("retry", message.Message.Data); + secondAttempt.Signal(); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { RedeliveryBackoff = _ => TimeSpan.FromMinutes(1), MaxAttempts = 3 }, cts.Token); + + await queue.SendAsync(new PreviewWorkItem { Data = "retry" }, cancellationToken: cts.Token); + await firstAttempt.WaitAsync(TimeSpan.FromSeconds(2)); + + // The retry is parked in the runtime store: the still-attached consumer must NOT get a second attempt until + // the dispatch pump drains the store. + await Task.Delay(250, cancellationToken); + Assert.Equal(1, attempts); + + Assert.Equal(1, await processor.RunDueOccurrencesAsync(DateTimeOffset.UtcNow.AddMinutes(2), cancellationToken: cancellationToken)); + await secondAttempt.WaitAsync(TimeSpan.FromSeconds(2)); + + } + + [Fact] + public async Task EnqueueAsync_ExceedingTransportMaxMessageBytes_ThrowsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + // The transport advertises an 8-byte maximum; the core must enforce it up front with a clear error rather than + // let an opaque broker rejection surface mid-send. + await using var transport = new BatchLimitTransport(maxBatchSize: 10, maxMessageBytes: 8); + await using var queue = new MessageBus(transport); + + await Assert.ThrowsAsync(async () => + await queue.SendAsync(new PreviewWorkItem { Data = "a payload well over eight bytes" }, cancellationToken: cancellationToken)); + } + + [Fact] + public async Task RejectAsync_RuntimeStoreRedelivery_AdvancesAttemptCountEachCycleAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var store = new InMemoryJobRuntimeStore(); + // BasicQueueTransport resets DeliveryCount to 1 on every (re)send and has no native redelivery delay, so each + // delayed reject re-schedules through the runtime store. The reconciled Attempts must keep advancing across + // redeliveries (1 -> 2 -> 3); a regression that bases the next attempt on the reset DeliveryCount would pin it at + // 2 and redeliver forever (never reaching MaxAttempts / dead-letter). + await using var transport = new BasicQueueTransport(); + await using var queue = new MessageBus(transport, new MessageBusOptions { RuntimeStore = store }); + var processor = CreateDispatchProcessor(store, transport); + var now = DateTimeOffset.UtcNow; + + await queue.SendAsync(new PreviewWorkItem { Data = "loop" }, cancellationToken: cancellationToken); + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + + for (int expectedAttempt = 1; expectedAttempt <= 3; expectedAttempt++) + { + var received = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(received); + Assert.Equal(expectedAttempt, received.Attempts); + Assert.Equal("loop", received.Message.Data); + + if (expectedAttempt < 3) + { + await received.RejectAsync(new RejectOptions { RedeliveryDelay = TimeSpan.FromMinutes(1) }, cancellationToken); + Assert.Equal(1, await processor.RunDueOccurrencesAsync(now.AddMinutes(expectedAttempt * 2), cancellationToken: cancellationToken)); + } + else + { + await received.CompleteAsync(cancellationToken); + } + } + } + + [Fact] + public async Task SendAsync_WithExpiredMessage_IsDeadLetteredNotDeliveredAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport); + + await queue.SendAsync(new PreviewWorkItem { Data = "expired" }, new MessageSendOptions { TimeToLive = TimeSpan.FromMilliseconds(-1) }, cancellationToken); + + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + var received = await collector.NextAsync(TimeSpan.FromMilliseconds(500), cancellationToken); + Assert.Null(received); + + var stats = await transport.GetStatsAsync(DestinationAddress.ForQueue("preview-work-item"), cancellationToken); + Assert.Equal(1, stats.Deadletter); + } + + + [Fact] + public async Task AddFoundatio_WithInMemoryMessagingAndJobs_RegistersAppFacingServices() + { + var services = new ServiceCollection(); + + services.AddFoundatio() + .Messaging.UseInMemory() + .Jobs.UseInMemoryRuntime(); + + await using var provider = services.BuildServiceProvider(); + + Assert.NotNull(provider.GetRequiredService()); + Assert.NotNull(provider.GetRequiredService()); + Assert.NotNull(provider.GetRequiredService()); + Assert.NotNull(provider.GetRequiredService()); + Assert.NotNull(provider.GetRequiredService()); + Assert.NotNull(provider.GetRequiredService()); + } + + [Fact] + public async Task AddFoundatio_WithRouting_RegistersRouterAndTopologyAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var services = new ServiceCollection(); + + services.AddFoundatio() + .Messaging.ConfigureRouting(r => r + .UseDefaultQueue("all-work") + .MapTopic("grouped-events", typeof(IGroupedWorkItem)) + .UseServiceIdentity("billing-service")) + .UseInMemory(); + + await using var provider = services.BuildServiceProvider(); + + var router = provider.GetRequiredService(); + Assert.Equal("all-work", router.ResolveRoute(new MessageRouteContext + { + MessageType = typeof(PreviewWorkItem), + Role = MessageRouteRole.QueueDestination + })); + Assert.Equal("grouped-events", router.ResolveRoute(new MessageRouteContext + { + MessageType = typeof(OtherWorkItem), + Role = MessageRouteRole.PubSubTopic + })); + + var topology = provider.GetRequiredService(); + var declarations = topology.GetDeclarations(); + Assert.Contains(declarations, d => d.Address.Role == DestinationRole.Queue && d.Address.Name == "all-work"); + Assert.Contains(declarations, d => d.Address.Role == DestinationRole.Topic && d.Address.Name == "grouped-events"); + Assert.Contains(declarations, d => d.Address.Role == DestinationRole.Subscription && d.Address.Name == "billing-service" && d.Address.Topic == "grouped-events"); + + await Assert.ThrowsAsync(async () => await topology.ValidateAsync(cancellationToken)); + await topology.EnsureAsync(cancellationToken); + await topology.ValidateAsync(cancellationToken); + } + + [Fact] + public async Task EnqueueAsync_WithRouteAttribute_UsesAttributedDestinationAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var queue = new MessageBus(new InMemoryMessageTransport()); + + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + Assert.Equal("routed-work", collector.Destination); // the [MessageRoute] attribute names the send destination + + await queue.SendAsync(new RoutedWorkItem { Data = "route" }, cancellationToken: cancellationToken); + + var received = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(received); + Assert.Equal("route", received.Message.Data); + await received.CompleteAsync(cancellationToken); + } + + [Fact] + public async Task StartConsumerAsync_WithSameKeyAndSameRegistration_SharesTheUnderlyingConsumerAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var queue = new MessageBus(new InMemoryMessageTransport()); + int handled = 0; + var received = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + Func, CancellationToken, Task> handler = (_, _) => + { + Interlocked.Increment(ref handled); + received.TrySetResult(); + return Task.CompletedTask; + }; + + // Registering the same key + handler + options twice is idempotent: both handles refer to the one underlying + // consumer, so a sent message is handled exactly once. + await using var first = await queue.SubscribeAsync(handler, new MessageSubscriptionOptions { Key = "shared" }, cancellationToken); + await using var second = await queue.SubscribeAsync(handler, new MessageSubscriptionOptions { Key = "shared" }, cancellationToken); + + Assert.Equal(first.Key, second.Key); + Assert.Equal(first.Destination, second.Destination); + + await queue.SendAsync(new PreviewWorkItem { Data = "once" }, cancellationToken: cancellationToken); + await received.Task.WaitAsync(TimeSpan.FromSeconds(10), cancellationToken); + await Task.Delay(250, cancellationToken); + Assert.Equal(1, Volatile.Read(ref handled)); + } + + [Fact] + public async Task StartConsumerAsync_WithSameKeyAndDifferentHandler_ThrowsAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var queue = new MessageBus(new InMemoryMessageTransport()); + + await using var first = await queue.SubscribeAsync((_, _) => Task.CompletedTask, new MessageSubscriptionOptions { Key = "shared" }, cancellationToken); + + await Assert.ThrowsAsync(async () => + await queue.SubscribeAsync((_, _) => Task.CompletedTask, new MessageSubscriptionOptions { Key = "shared" }, cancellationToken)); + } + + [Fact] + public async Task ReceiveAsync_WithGroupedInterfaceRoute_ReturnsRawMessagesAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var routing = new MessageRoutingOptionsBuilder() + .MapQueue("grouped-work", typeof(IGroupedWorkItem)) + .Build(); + await using var queue = new MessageBus(new InMemoryMessageTransport(), new MessageBusOptions { Router = new DefaultMessageRouter(routing) }); + + await queue.SendBatchAsync(new object[] + { + new PreviewWorkItem { Data = "one" }, + new OtherWorkItem { Data = "two" } + }, cancellationToken: cancellationToken); + + await using var collector = await MessageCollector.StartAsync(queue, routeType: typeof(IGroupedWorkItem), cancellationToken: cancellationToken); + var first = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + var second = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + + Assert.NotNull(first); + Assert.NotNull(second); + Assert.NotEmpty(first.Body.ToArray()); + Assert.Equal(typeof(PreviewWorkItem).FullName, first.MessageType); + Assert.Equal(typeof(OtherWorkItem).FullName, second.MessageType); + + await first.CompleteAsync(cancellationToken); + await second.CompleteAsync(cancellationToken); + } + + [Fact] + public async Task StartConsumerAsync_WithGroupedInterfaceRoute_DeserializesConcreteTypeAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var routing = new MessageRoutingOptionsBuilder() + .MapQueue("grouped-work", typeof(IGroupedWorkItem)) + .Build(); + await using var queue = new MessageBus(new InMemoryMessageTransport(), new MessageBusOptions { Router = new DefaultMessageRouter(routing) }); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + + var received = new ConcurrentDictionary(); + var signal = new AsyncCountdownEvent(2); + + // An interface-typed consumer receives the concrete payload (assignable to the interface), not raw bytes — + // the core resolves the concrete type from the message-type header and deserializes that. + await using var consumer = await queue.SubscribeAsync((message, _) => + { + string? data = message.Message switch + { + PreviewWorkItem p => p.Data, + OtherWorkItem o => o.Data, + _ => null + }; + received[message.Message.GetType()] = data; + signal.Signal(); + return Task.CompletedTask; + }, cancellationToken: cts.Token); + + await queue.SendBatchAsync(new object[] + { + new PreviewWorkItem { Data = "one" }, + new OtherWorkItem { Data = "two" } + }, cancellationToken: cts.Token); + + await signal.WaitAsync(TimeSpan.FromSeconds(2)); + + Assert.Equal("one", received[typeof(PreviewWorkItem)]); + Assert.Equal("two", received[typeof(OtherWorkItem)]); + } + + [Fact] + public async Task ReceiveAsync_WithDefaultQueueRoute_ReturnsRawMessageAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + var routing = new MessageRoutingOptionsBuilder() + .UseDefaultQueue("all-work") + .Build(); + await using var queue = new MessageBus(new InMemoryMessageTransport(), new MessageBusOptions { Router = new DefaultMessageRouter(routing) }); + + await queue.SendAsync(new PreviewWorkItem { Data = "global" }, cancellationToken: cancellationToken); + + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + var received = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + + Assert.NotNull(received); + Assert.Equal(typeof(PreviewWorkItem).FullName, received.MessageType); + await received.CompleteAsync(cancellationToken); + } + + + private static async Task WaitForCompletedAsync(InMemoryMessageTransport transport, string destination, CancellationToken cancellationToken) + { + var address = DestinationAddress.ForQueue(destination); + var deadline = DateTimeOffset.UtcNow.AddSeconds(2); + while (DateTimeOffset.UtcNow < deadline) + { + var stats = await transport.GetStatsAsync(address, cancellationToken); + if (stats.Completed == 1) + return; + + await Task.Delay(TimeSpan.FromMilliseconds(10), cancellationToken); + } + + var finalStats = await transport.GetStatsAsync(address, cancellationToken); + Assert.Equal(1, finalStats.Completed); + } + + // Pull-style test helper over the subscription API: collects manually-acked contexts so tests can inspect and + // settle deliveries explicitly, now that the bus surface is subscription-only. + private sealed class MessageCollector : IAsyncDisposable where T : class + { + private readonly Channel> _received = Channel.CreateUnbounded>(); + private IMessageSubscription _subscription = null!; + + public static async Task> StartAsync(IMessageBus bus, string? destination = null, CancellationToken cancellationToken = default) + { + var collector = new MessageCollector(); + collector._subscription = await bus.SubscribeAsync((context, _) => + { + collector._received.Writer.TryWrite(context); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { AckMode = AckMode.Manual, Destination = destination }, cancellationToken); + return collector; + } + + public string Destination => _subscription.Destination; + + public async Task?> NextAsync(TimeSpan maxWait, CancellationToken cancellationToken = default) + { + // WaitToReadAsync + TryRead (not ReadAsync + WaitAsync): a timed-out WaitAsync abandons its ReadAsync, + // which would silently consume the next item. + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(maxWait); + try + { + while (await _received.Reader.WaitToReadAsync(cts.Token)) + { + if (_received.Reader.TryRead(out var context)) + return context; + } + + return null; + } + catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested) + { + return null; + } + } + + public ValueTask DisposeAsync() => _subscription.DisposeAsync(); + } + + private sealed class MessageCollector : IAsyncDisposable + { + private readonly Channel _received = Channel.CreateUnbounded(); + private IMessageSubscription _subscription = null!; + + public static async Task StartAsync(IMessageBus bus, Type? routeType = null, string? destination = null, CancellationToken cancellationToken = default) + { + var collector = new MessageCollector(); + collector._subscription = await bus.SubscribeAsync((context, _) => + { + collector._received.Writer.TryWrite(context); + return Task.CompletedTask; + }, new MessageSubscriptionOptions { AckMode = AckMode.Manual, RouteType = routeType, Destination = destination }, cancellationToken); + return collector; + } + + public async Task NextAsync(TimeSpan maxWait, CancellationToken cancellationToken = default) + { + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(maxWait); + try + { + while (await _received.Reader.WaitToReadAsync(cts.Token)) + { + if (_received.Reader.TryRead(out var context)) + return context; + } + + return null; + } + catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested) + { + return null; + } + } + + public ValueTask DisposeAsync() => _subscription.DisposeAsync(); + } + + [Fact] + public async Task StartConsumerAsync_MultipleTypesOnOneDestination_DispatchByTypeAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var queue = new MessageBus(new InMemoryMessageTransport()); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(10)); + + var aReceived = new List(); + var bReceived = new List(); + var aSignal = new AsyncCountdownEvent(1); + var bSignal = new AsyncCountdownEvent(1); + + await using var consumerA = await queue.SubscribeAsync((message, _) => + { + lock (aReceived) + aReceived.Add(message.Message.Data); + aSignal.Signal(); + return Task.CompletedTask; + }, cancellationToken: cts.Token); + + await using var consumerB = await queue.SubscribeAsync((message, _) => + { + lock (bReceived) + bReceived.Add(message.Message.Data); + bSignal.Signal(); + return Task.CompletedTask; + }, cancellationToken: cts.Token); + + // Both types route to the same destination, so they share one underlying receive loop that dispatches by type. + Assert.Equal(consumerA.Source, consumerB.Source); + + await queue.SendAsync(new SharedAWorkItem { Data = "a" }, cancellationToken: cts.Token); + await queue.SendAsync(new SharedBWorkItem { Data = "b" }, cancellationToken: cts.Token); + + await aSignal.WaitAsync(TimeSpan.FromSeconds(2)); + await bSignal.WaitAsync(TimeSpan.FromSeconds(2)); + + Assert.Equal(new[] { "a" }, aReceived); + Assert.Equal(new[] { "b" }, bReceived); + } + + [Fact] + public async Task StartConsumerAsync_UnmatchedType_DeadLettersAndKeepsConsumingAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport, new MessageBusOptions { RetryPolicy = new RetryPolicy { UnmatchedMaxAttempts = 3 } }); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(20)); + + var aSignal = new AsyncCountdownEvent(1); + await using var consumerA = await queue.SubscribeAsync((_, _) => + { + aSignal.Signal(); + return Task.CompletedTask; + }, cancellationToken: cts.Token); + + // SharedBWorkItem routes to the same destination but has no registered consumer on this node. + await queue.SendAsync(new SharedBWorkItem { Data = "orphan" }, cancellationToken: cts.Token); + + // It is retried and finally dead-lettered as "no-handler" once the configured unmatched budget is exhausted. + for (int i = 0; i < 400; i++) + { + if ((await transport.GetStatsAsync(DestinationAddress.ForQueue("shared-demux"), cts.Token)).Deadletter == 1) + break; + await Task.Delay(TimeSpan.FromMilliseconds(25), cts.Token); + } + + Assert.Equal(1, (await transport.GetStatsAsync(DestinationAddress.ForQueue("shared-demux"), cts.Token)).Deadletter); + + // The loop survived the unmatched message and keeps consuming the type it does handle. + await queue.SendAsync(new SharedAWorkItem { Data = "ok" }, cancellationToken: cts.Token); + await aSignal.WaitAsync(TimeSpan.FromSeconds(2)); + } + + [Fact] + public async Task RejectAsync_Terminal_WithoutNativeDeadLetter_SendsToConfiguredDestinationAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new NoDeadLetterTransport(); + await using var queue = new MessageBus(transport, new MessageBusOptions { RetryPolicy = new RetryPolicy { DeadLetterDestination = "preview-dead-letter" } }); + + await queue.SendAsync(new PreviewWorkItem { Data = "bad" }, cancellationToken: cancellationToken); + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + var message = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(message); + + await message.RejectAsync(new RejectOptions { Terminal = true, Reason = "validation" }, cancellationToken); + + // The transport has no native dead-letter sink, so core routes the terminal message to the configured destination. + await using var deadCollector = await MessageCollector.StartAsync(queue, destination: "preview-dead-letter", cancellationToken: cancellationToken); + var dead = await deadCollector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(dead); + Assert.Equal("validation", dead.Headers.GetValueOrDefault(KnownHeaders.DeadLetterReason)); + } + + [Fact] + public async Task RejectAsync_Terminal_WithoutNativeDeadLetterOrConfig_DerivesDeadLetterDestinationAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new NoDeadLetterTransport(); + // No DeadLetterDestination configured: the terminal message must be parked at "{source}.deadletter", not dropped. + await using var queue = new MessageBus(transport); + + await queue.SendAsync(new PreviewWorkItem { Data = "bad" }, cancellationToken: cancellationToken); + await using var collector = await MessageCollector.StartAsync(queue, cancellationToken: cancellationToken); + var message = await collector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(message); + + await message.RejectAsync(new RejectOptions { Terminal = true, Reason = "validation", Exception = new InvalidOperationException("boom") }, cancellationToken); + + await using var deadCollector = await MessageCollector.StartAsync(queue, destination: "preview-work-item.deadletter", cancellationToken: cancellationToken); + var dead = await deadCollector.NextAsync(TimeSpan.FromSeconds(2), cancellationToken); + Assert.NotNull(dead); + Assert.Equal("validation", dead.Headers.GetValueOrDefault(KnownHeaders.DeadLetterReason)); + Assert.Equal(typeof(InvalidOperationException).FullName, dead.Headers.GetValueOrDefault(KnownHeaders.DeadLetterExceptionType)); + Assert.Equal("preview-work-item", dead.Headers.GetValueOrDefault(KnownHeaders.DeadLetterOriginalDestination)); + } + + [Fact] + public async Task RetryPolicy_BackoffOnTransportWithoutDelaySupport_DegradesToImmediateRedeliveryAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + // BasicQueueTransport has no native redelivery delay and no runtime store is registered, so the default + // backoff curve (10s+ after the second attempt) cannot be honored — the policy retry must degrade to + // immediate redelivery rather than failing the settle, and still reach dead-letter after MaxAttempts. + await using var transport = new BasicQueueTransport(); + await using var queue = new MessageBus(transport); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(15)); + int attempts = 0; + + await using var consumer = await queue.SubscribeAsync((_, _) => + { + Interlocked.Increment(ref attempts); + throw new InvalidOperationException("always fails"); + }, new MessageSubscriptionOptions { MaxAttempts = 3 }, cts.Token); + + await queue.SendAsync(new PreviewWorkItem { Data = "doomed" }, cancellationToken: cts.Token); + + // All three attempts happen without a 10s stall, ending in the transport's native dead-letter sink. + var stats = await transport.GetStatsAsync(DestinationAddress.ForQueue("preview-work-item"), cts.Token); + long deadline = Environment.TickCount64 + 10_000; + while (stats.Deadletter == 0 && Environment.TickCount64 < deadline) + { + await Task.Delay(25, cts.Token); + stats = await transport.GetStatsAsync(DestinationAddress.ForQueue("preview-work-item"), cts.Token); + } + + Assert.Equal(1, stats.Deadletter); + Assert.Equal(3, Volatile.Read(ref attempts)); + } + + [Fact] + public async Task StartConsumerAsync_UsesDefaultRetryPolicyMaxAttempts_WhenConsumerDoesNotOverrideAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + await using var transport = new InMemoryMessageTransport(); + await using var queue = new MessageBus(transport, new MessageBusOptions { RetryPolicy = new RetryPolicy { MaxAttempts = 2 } }); + using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + cts.CancelAfter(TimeSpan.FromSeconds(20)); + + int attempts = 0; + await using var consumer = await queue.SubscribeAsync((_, _) => + { + Interlocked.Increment(ref attempts); + throw new InvalidOperationException("always fails"); + }, cancellationToken: cts.Token); // no per-consumer MaxAttempts -> default RetryPolicy (2) + + await queue.SendAsync(new PreviewWorkItem { Data = "x" }, cancellationToken: cts.Token); + + for (int i = 0; i < 400; i++) + { + if ((await transport.GetStatsAsync(DestinationAddress.ForQueue("preview-work-item"), cts.Token)).Deadletter == 1) + break; + await Task.Delay(TimeSpan.FromMilliseconds(25), cts.Token); + } + + Assert.Equal(1, (await transport.GetStatsAsync(DestinationAddress.ForQueue("preview-work-item"), cts.Token)).Deadletter); + Assert.Equal(2, attempts); + } + + [Fact] + public async Task DisposeAsync_RespectsTransportOwnershipAsync() + { + var cancellationToken = TestContext.Current.CancellationToken; + + // Non-owning client (shared transport): disposing the client leaves the transport usable. + var shared = new InMemoryMessageTransport(); + var nonOwning = new MessageBus(shared, new MessageBusOptions { OwnsTransport = false }); + await nonOwning.DisposeAsync(); + await shared.SendAsync(DestinationAddress.ForQueue("still-alive"), [new TransportMessage { Body = ReadOnlyMemory.Empty }], new TransportSendOptions(), cancellationToken); + await shared.DisposeAsync(); + + // Owning client (default): disposing the client disposes the transport. + var owned = new InMemoryMessageTransport(); + var owning = new MessageBus(owned); + await owning.DisposeAsync(); + await Assert.ThrowsAsync(async () => + await owned.SendAsync(DestinationAddress.ForQueue("dead"), [new TransportMessage { Body = ReadOnlyMemory.Empty }], new TransportSendOptions(), cancellationToken)); + } + + [Fact] + public async Task DiBuiltClients_ShareTransport_DisposedExactlyOnceAsync() + { + var transport = new DisposeCountingTransport(); + var services = new ServiceCollection(); + services.AddFoundatio().Messaging.UseTransport(transport); + await using var provider = services.BuildServiceProvider(); + + _ = provider.GetRequiredService(); + + await provider.DisposeAsync(); + + // The container owns the shared transport singleton; the bus does not dispose it, so it is disposed once. + Assert.Equal(1, transport.DisposeCount); + } + + private static JobScheduleProcessor CreateDispatchProcessor(IJobRuntimeStore store, IMessageTransport transport) + { + var serviceProvider = new ServiceCollection().BuildServiceProvider(); + var worker = new JobWorker(store, serviceProvider, nodeId: "node-a"); + return new JobScheduleProcessor(new InMemoryJobScheduler(), store, worker, nodeId: "node-a", transport: transport); + } + + [MessageRoute("routed-work")] + private sealed class RoutedWorkItem + { + public string? Data { get; set; } + } + + [MessageRoute("shared-demux")] + private sealed class SharedAWorkItem + { + public string? Data { get; set; } + } + + [MessageRoute("shared-demux")] + private sealed class SharedBWorkItem + { + public string? Data { get; set; } + } + + private interface IGroupedWorkItem + { + } + + private sealed class PreviewWorkItem : IGroupedWorkItem + { + public string? Data { get; set; } + } + + private sealed class OtherWorkItem : IGroupedWorkItem + { + public string? Data { get; set; } + } + + private sealed class BatchLimitTransport : IMessageTransport, ITransportInfo + { + public BatchLimitTransport(int maxBatchSize, long? maxMessageBytes = null) + { + MaxBatchSize = maxBatchSize; + MaxMessageBytes = maxMessageBytes; + } + + public List SendBatchSizes { get; } = new(); + public DeliveryGuarantee DeliveryGuarantee => DeliveryGuarantee.AtLeastOnce; + public IReadOnlySet SupportedRoles => new HashSet { DestinationRole.Queue }; + public int? MaxBatchSize { get; } + public long? MaxMessageBytes { get; } + + public TransportCapabilities GetCapabilities(DestinationRole role) => + new() { Ordering = OrderingGuarantee.Fifo, MaxBatchSize = MaxBatchSize, MaxMessageBytes = MaxMessageBytes }; + + public Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + SendBatchSizes.Add(messages.Count); + var items = new SendItemResult[messages.Count]; + for (int i = 0; i < messages.Count; i++) + items[i] = new SendItemResult { MessageId = messages[i].MessageId ?? Guid.NewGuid().ToString("N") }; + + return Task.FromResult(new SendResult { Items = items }); + } + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + public ValueTask DisposeAsync() => ValueTask.CompletedTask; + } + + private sealed class CappedDelayTransport : IMessageTransport, ISupportsPull, ITransportInfo + { + private readonly Queue _entries = new(); + + public CappedDelayTransport(TimeSpan? maxDeliveryDelay) + { + MaxDeliveryDelay = maxDeliveryDelay; + } + + public TimeSpan? MaxDeliveryDelay { get; } + public int SendCount { get; private set; } + public TransportSendOptions? LastSendOptions { get; private set; } + + public DeliveryGuarantee DeliveryGuarantee => DeliveryGuarantee.AtLeastOnce; + public IReadOnlySet SupportedRoles => new HashSet { DestinationRole.Queue }; + + public TransportCapabilities GetCapabilities(DestinationRole role) => + new() { DelayedDelivery = true, MaxDeliveryDelay = MaxDeliveryDelay }; + + public Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + SendCount += messages.Count; + LastSendOptions = options; + var items = new SendItemResult[messages.Count]; + for (int i = 0; i < messages.Count; i++) + { + string id = messages[i].MessageId ?? Guid.NewGuid().ToString("N"); + _entries.Enqueue(new TransportEntry { Id = id, Destination = destination, Body = messages[i].Body, Headers = messages[i].Headers, Receipt = new Receipt() }); + items[i] = new SendItemResult { MessageId = id }; + } + + return Task.FromResult(new SendResult { Items = items }); + } + + public Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct) + { + return Task.FromResult>(_entries.Count > 0 ? [_entries.Dequeue()] : []); + } + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + public ValueTask DisposeAsync() => ValueTask.CompletedTask; + } + + // A minimal multi-destination pull transport with NO native dead-letter sink, used to prove core-managed + // dead-lettering routes terminal messages to the configured RetryPolicy.DeadLetterDestination. + private sealed class NoDeadLetterTransport : IMessageTransport, ISupportsPull + { + private readonly ConcurrentDictionary> _queues = new(StringComparer.Ordinal); + + public Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + { + var queue = _queues.GetOrAdd(destination.Key, _ => new ConcurrentQueue()); + var items = new SendItemResult[messages.Count]; + for (int i = 0; i < messages.Count; i++) + { + string id = messages[i].MessageId ?? Guid.NewGuid().ToString("N"); + queue.Enqueue(new TransportEntry { Id = id, Destination = destination, Body = messages[i].Body, Headers = messages[i].Headers, Receipt = new Receipt() }); + items[i] = new SendItemResult { MessageId = id }; + } + + return Task.FromResult(new SendResult { Items = items }); + } + + public Task> ReceiveAsync(DestinationAddress source, ReceiveRequest request, CancellationToken ct) + { + if (_queues.TryGetValue(source.Key, out var queue) && queue.TryDequeue(out var entry)) + return Task.FromResult>([entry]); + + return Task.FromResult>([]); + } + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) + { + _queues.GetOrAdd(entry.Destination.Key, _ => new ConcurrentQueue()).Enqueue(entry with { DeliveryCount = entry.DeliveryCount + 1 }); + return Task.CompletedTask; + } + + public ValueTask DisposeAsync() => ValueTask.CompletedTask; + } + + // Counts dispose calls without an idempotency guard, so a double-dispose (the bug item 6 fixes) would show as > 1. + private sealed class DisposeCountingTransport : IMessageTransport + { + public int DisposeCount { get; private set; } + + public Task SendAsync(DestinationAddress destination, IReadOnlyList messages, TransportSendOptions options, CancellationToken ct = default) + => Task.FromResult(new SendResult { Items = Array.Empty() }); + + public Task CompleteAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + public Task AbandonAsync(TransportEntry entry, CancellationToken ct = default) => Task.CompletedTask; + + public ValueTask DisposeAsync() + { + DisposeCount++; + return ValueTask.CompletedTask; + } + } +} \ No newline at end of file diff --git a/tests/Foundatio.Tests/Utility/ResiliencePolicyTests.cs b/tests/Foundatio.Tests/Utility/ResiliencePolicyTests.cs index 95ac75e3e..16671bc8e 100644 --- a/tests/Foundatio.Tests/Utility/ResiliencePolicyTests.cs +++ b/tests/Foundatio.Tests/Utility/ResiliencePolicyTests.cs @@ -5,7 +5,7 @@ using System.Threading.Tasks; using Foundatio.Caching; using Foundatio.Lock; -using Foundatio.Messaging; +using Foundatio.Messaging.Legacy; using Foundatio.Resilience; using Foundatio.Utility; using Foundatio.Xunit;