Wait for in-flight health check publishers in HealthCheckPublisherHostedService.StopAsync - #68983
Conversation
…tedService.StopAsync
|
Thanks for your PR, @SergioAlmeida29. Someone from the team will get assigned to your PR shortly and we'll get it reviewed. |
There was a problem hiding this comment.
🟡 Changes recommended
There is a shutdown race where a timer callback can start a new tracked RunAsync after StopAsync snapshots an empty set and returns, allowing publish work to outlive shutdown.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR updates HealthCheckPublisherHostedService.StopAsync to await timer-triggered health check publishing work that’s already in flight during shutdown, preventing publishers from continuing after the host has stopped.
Changes:
- Track each timer-driven
RunAsyncinvocation in a shared set and await remaining tasks duringStopAsync, bounded by the caller’s cancellation token/shutdown timeout. - Replace the timer’s
async voidcallback pattern with a fire-and-forgetTaskmethod that records/removes the in-flight operation. - Add regression tests validating
StopAsyncwaits for an in-flight publisher, and that a canceled stop token does not wait.
File summaries
| File | Description |
|---|---|
| src/HealthChecks/HealthChecks/src/HealthCheckPublisherHostedService.cs | Track timer-triggered RunAsync tasks and await in-flight publishes during StopAsync. |
| src/HealthChecks/HealthChecks/test/HealthCheckPublisherHostedServiceTest.cs | Add regression tests covering shutdown waiting and cancellation-bounded stop behavior. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
Closes the shutdown race where a timer callback that reaches RunTimerCallbackAsync after StopAsync snapshots _runningTasks could still start a RunAsync that outlives StopAsync.
0187e1b to
aa14098
Compare
| private readonly IHealthCheckPublisher[] _publishers; | ||
| private List<Timer>? _timers; | ||
|
|
||
| private readonly object _runningTasksLock = new object(); |
There was a problem hiding this comment.
| private readonly object _runningTasksLock = new object(); | |
| private readonly Lock _runningTasksLock = new(); |
or does this build for older targets w/o Lock-type?
Fixes #67304
Problem
HealthCheckPublisherHostedService.StopAsyncdisposes its timers but doesn't wait for a health-check publish that is already running, soIHealthCheckPublisher.PublishAsynccan keep running after the host has stopped.The timer callback ran
RunAsyncas fire-and-forget — anasync voidlambda bound toTimerCallback— so nothing tracked the task.Timer.Dispose()(and evenDispose(WaitHandle)/DisposeAsync()) only wait for the synchronous portion of the callback, which ends at the firstawait.Change
Taskreturned by each timer-drivenRunAsyncin a set.StopAsyncawaits the in-flight tasks after disposing the timers, bounded by itsCancellationToken(the host links this toShutdownTimeout). MirrorsBackgroundService.StopAsync:#if NETusesTask.WaitAsync(token)withConfigureAwaitOptions.SuppressThrowing; older TFMs useTask.WhenAny+token.Register.Behavior note
StopAsyncmay now block up to the host shutdown timeout while a publish drains. This matches theIHostedServicecontract andBackgroundService, and is not an API or binary breaking change.Tests
Two regression tests driven by the real timer.
StopAsync_WaitsForInFlightPublisher_BeforeReturningfails without the fix (StopAsynccompletes while the publish is still parked).cc @danroth27 @BrennanConroy