Make server shutdown awaitable and report abnormal termination

[leynos]

Summary

Give WireframeServer an awaitable shutdown contract: an explicit stop that waits for the
accept loops to drain, a retained JoinHandle so abnormal termination is observed rather
than swallowed, and a typed error distinguishing a crash from a clean stop.

This is the server-side mirror of the pattern ADR 013 sections 9 and 10 adopt for the
client pool.

Problem

Today the only way to stop a WireframeServer from outside is to resolve the shutdown
future passed to run_with_shutdown, or to abort the task. #656 makes an aborted server
eventually release its listener, but the caller still cannot observe when that has
happened:

• run_with_shutdown owns its CancellationToken and TaskTracker as locals
  (src/server/runtime.rs:157-158), so no caller can cancel and then drain;
• awaiting the supervisor's JoinHandle after an abort yields
  Err(JoinError::cancelled), which says nothing about whether the accept loops have
  exited;
• a supervisor panic is visible only if the caller happens to retain and inspect the
  JoinHandle;
• there is no error that distinguishes "stopped cleanly" from "the server task died".

The practical consequence is that a caller that needs a hard guarantee — a test harness
reclaiming a port, a supervisor restarting a listener, an orchestrator rolling a deploy —
has to poll the socket and hope.

Prior art in this repository

ADR 013 section 9 ("Make shutdown awaitable") and section 10 ("Detect and report scheduler
failure") already settle this shape for WireframeClientPool:

• close delivers a shutdown command, waits for acknowledgement and task completion, then
  releases resources;
• shutdown delivery must be guaranteed non-blocking — reserved mailbox capacity, a
  dedicated channel, or a CancellationToken — so a saturated component cannot deadlock
  its own shutdown;
• the task's JoinHandle is retained so a panic is observed rather than swallowed;
• abnormal termination maps to a dedicated error variant, distinct from clean shutdown, so
  callers can programmatically tell a crash from a planned stop;
• abnormal termination emits an error-level structured log event.

This issue applies the same five properties to the server. It should not invent a second
vocabulary: where ADR 013 names a concept, reuse the name.

Proposed shape

Exact API to be settled during design, but it must satisfy the ADR 013 properties. A
sketch:

/// Returned alongside the running server; cloneable control, single-owner state.
pub struct ServerShutdown { /* cancellation handle + drain acknowledgement */ }

impl ServerShutdown {
    /// Non-blocking. Requests that accept loops stop.
    pub fn stop(&self);
    /// Waits until every accept loop has exited and the listener is released.
    pub async fn drained(&self) -> Result<(), ServerError>;
}

with run_with_shutdown either accepting a caller-supplied CancellationToken or
returning the handle. Both satisfy ADR 011 R6, which explicitly retains "legitimate shared
handles" — a cloneable control handle over single-owner state is the R6 shape, and R5's
actor guidance applies if the drain state acquires coupled invariants.

Abnormal termination gets a dedicated ServerError variant, distinct from Bind and from
whatever #642 adds for application build and preparation failure.

Sequencing

This restructures run_with_shutdown, and so does #642. They should not race.

• Cancel server accept loops when the run_with_shutdown future is dropped #656 first — independent, two lines, fixes the live leak, and this issue's drain
  guarantee builds on the cancellation it installs.
• Prepare the application before server readiness and share it across connection tasks #642 next — establishes the prepared-application startup sequence and the typed
  startup error model that this issue's termination error must sit alongside.
• This issue after Prepare the application before server readiness and share it across connection tasks #642, so the error model is extended once rather than twice.

ADR disposition

ADR 013 governs the client pool, so it cannot be cited directly as authority for a server
change. This issue therefore needs either:

• a new ADR recording the server runtime's shutdown-ownership contract as the symmetric
  application of ADR 013 sections 9 and 10; or
• an agreed amendment generalizing those sections to both runtimes.

The next free ADR number is 014, ADRs 011 to 013 being taken by #653. Record the choice
before implementation begins.

Acceptance criteria

• A caller can request shutdown and then await a guarantee that every accept loop has
  exited and the listener is released.
• Shutdown delivery is non-blocking and cannot deadlock against a saturated server.
• The server task's JoinHandle is retained so a panic is observed, not swallowed.
• Abnormal termination maps to a dedicated ServerError variant, distinct from clean
  shutdown and from the Prepare the application before server readiness and share it across connection tasks #642 startup variants, preserving the source error.
• Abnormal termination emits an error-level structured log event.
• Repeated stop requests are safe and converge to one drain.
• run() behaviour and graceful-drain semantics for in-flight connections are unchanged.
• The ADR disposition above is recorded before merge.

Tests

• Request shutdown, await the drain, then assert the address refuses connections
  immediately — with no polling and no sleep.
• Request shutdown on a server with an in-flight connection and assert the existing
  graceful-drain semantics are preserved.
• Panic inside the supervisor and assert the dedicated error variant plus the log event.
• Issue several concurrent stop requests and assert one drain and one terminal outcome.
• Assert shutdown delivery succeeds while the server is saturated.

Non-goals

• Cancelling in-flight connection tasks, or otherwise changing graceful drain.
• Application preparation and startup error modelling, which is Prepare the application before server readiness and share it across connection tasks #642.
• Restarting a failed server; ADR 013 section 10 sets the precedent that automatic restart
  is out of scope.
• The wireframe_testing lifecycle harness, which consumes this.

References

• Epic Epic: Align Tokio task boundaries with ownership boundaries #635
• ADR 013 proposal ADR 013 proposal: Single-owner client-pool scheduler and slot graph #638 — sections 9 and 10 are the pattern this mirrors
• ADR 011 proposal ADR 011 proposal: Runtime ownership and task-lifetime boundaries #636 — R5 and R6 govern the handle's ownership shape
• Cancel server accept loops when the run_with_shutdown future is dropped #656 — prerequisite; makes drop cancel the workers
• Prepare the application before server readiness and share it across connection tasks #642 — sequencing dependency; establishes the typed startup error model
• Replace spawn-on-demand pool servicing with a persistent scheduler actor #647 — the client-side implementation of the same pattern
• Publish ownership ADRs and add regression guidance after implementation #649 — final verification record
• src/server/runtime.rs, src/server/error.rs

[leynos]

@CodeRabbit plan

[coderabbitai]

Coding Plan

Summary

• Add a cloneable ServerShutdown handle over single-owner state, with a non-blocking stop() backed by CancellationToken and an awaitable drained() backed by a watch channel, following ADR 011 R5/R6 and ADR 013 sections 9 and 10.
• Refactor the supervisor core so a new spawn method can run it as a Tokio task, retain its JoinHandle in an observer, and record the terminal outcome.
• Add one dedicated ServerError variant plus an error-level metric and tracing event for abnormal termination, distinct from clean shutdown and from the #642 startup variants.
• Record the decision in ADR-014, verify the contract with harness-based tests, and document the new contract in the developer guide.

Design Choices

Design Choice 1: Choose the shutdown-control API shape

Options Considered:

1. Add a new method that spawns the supervisor onto Tokio, retains its JoinHandle, and returns a cloneable ServerShutdown. Leave run_with_shutdown and run() unchanged externally.
2. Change run_with_shutdown to accept a caller-supplied CancellationToken, and add a separate drain-await mechanism.

Chosen Option: 1

Rationale: The ticket allows either a returned handle or a caller-supplied CancellationToken. run_with_shutdown consumes self and awaits inline, so it cannot both return a handle and keep its current signature. A new method preserves run() and the existing tests, retains the JoinHandle needed to observe a panic, and matches the "returned alongside the running server" sketch and the ADR 011 R6 handle shape.

Design Choice 2: Choose how the drain outcome is shared and how the abnormal-termination error preserves its source

Options Considered:

1. Broadcast a small Clone-able terminal descriptor through a watch channel. Capture the panic payload as a String and carry it in the new ServerError variant.
2. Store the JoinHandle behind a Mutex<Option<...>>, let the first drained() caller await it, and let later callers read a stored marker.

Chosen Option: 1

Rationale: drained() takes &self and can be called by several cloned handles, but ServerError is not Clone and a panic yields a JoinError rather than an io::Error. A watch of a Clone-able terminal descriptor lets every concurrent drained() caller converge to one outcome without cloning ServerError, and the captured panic message preserves the diagnostic source in a lightweight form.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Record the decision and extend the error and observability model

This phase records the governing ADR and adds the typed error and the error-level observability that abnormal termination needs. Complete these tasks first, because the later runtime work depends on the new error variant and the recorded contract.

Task 1: Write ADR-014 for the server shutdown-ownership contract

Record the server-side shutdown contract as the symmetric application of ADR 013 sections 9 and 10.

• Create docs/adr-014-server-shutdown-ownership.md.
• Follow the section order used by ADR 012 and ADR 013: Title, Status, Date, Context and problem statement, Traceability, Decision drivers, Options considered (with a captioned comparison table), Decision outcome (with captioned figures where useful), Consequences (Positive/Negative), Rejected shortcuts, Migration plan, Verification, Outstanding decisions, References.
• Set ## Status to Proposed with the "First proposed on in issue #N" note.
• State that ADR-014 complements ADR 013 and applies its sections 9 and 10 to the server runtime. State that it does not supersede ADR 011, 012, or 013.
• Cite ADR 011 R5 and R6 as the ownership rules for the new handle. Record that the handle is a cloneable control over single-owner state.
• Reuse the ADR 013 vocabulary. Do not invent a second vocabulary for the same concepts.
• Use en-GB-oxendict spelling throughout.

Task 2: Add a dedicated `ServerError` variant for abnormal termination

Add a typed error that distinguishes a crashed supervisor from a clean stop.

• Update src/server/error.rs.
• Add one #[non_exhaustive]-compatible variant, for example AbnormalTermination, that carries the captured panic message as a String.
• Keep the variant distinct from Bind, from Accept, and from any startup variants that #642 adds.
• Give the variant a clear #[error(...)] message that includes the captured payload.

Task 3: Add an error-level metric and tracing event for abnormal termination

Report abnormal termination through the established observability pattern.

• Update src/metrics.rs. Add a counter constant, for example SERVER_SUPERVISOR_ABNORMAL_TERMINATIONS with value "wireframe_server_supervisor_abnormal_terminations_total". Document it with the /// ... doc comment and the plaintext HELP/TYPE block that the other metrics use.
• Add a pub(crate) fn inc_server_supervisor_abnormal_termination(...) with the paired #[cfg(feature = "metrics")] and #[cfg(not(feature = "metrics"))] twin, matching the existing increment functions.
• Add a record helper near the existing record_supervisor_cancellation logic in src/server/runtime.rs. Emit tracing::error!(event = "server_supervisor_abnormal_termination", ...) and call the increment function from the same helper.
• Keep any label bounded and low cardinality. Do not put the panic message into a metric label.

🤖 Prompt for AI agents

Implement Phase 1 of the awaitable server shutdown work: record the governing
ADR and add the typed error plus error-level observability for abnormal
termination.

- Create `docs/adr-014-server-shutdown-ownership.md`. Follow the section order
used by ADR 012 and ADR 013: Title, Status, Date, Context and problem statement,
Traceability, Decision drivers, Options considered (with a captioned comparison
table), Decision outcome (with captioned figures where useful), Consequences
(Positive/Negative), Rejected shortcuts, Migration plan, Verification,
Outstanding decisions, References. Set `## Status` to `Proposed` with a "First
proposed on <date> in issue `#N`" note. State that ADR-014 complements ADR 013 by
applying its sections 9 and 10 to the server runtime, and that it does not
supersede ADR 011, 012, or 013. Cite ADR 011 R5 and R6 as the ownership rules
for the new handle, and record the handle as a cloneable control over
single-owner state. Reuse ADR 013 vocabulary rather than inventing new terms.
Write in en-GB-oxendict spelling.

- In `src/server/error.rs`, add one new `#[non_exhaustive]`-compatible
`ServerError` variant, for example `AbnormalTermination`, carrying the captured
panic message as a `String`. Keep it distinct from `Bind`, `Accept`, and any
startup variants added by `#642`. Give it a clear `#[error(...)]` message that
includes the captured payload.

- In `src/metrics.rs`, add a counter constant, for example
`SERVER_SUPERVISOR_ABNORMAL_TERMINATIONS`, with value
`"wireframe_server_supervisor_abnormal_terminations_total"`, documented with the
`///` doc comment and `plaintext` HELP/TYPE block matching existing metrics. Add
`pub(crate) fn inc_server_supervisor_abnormal_termination(...)` with the paired
`#[cfg(feature = "metrics")]` / `#[cfg(not(feature = "metrics"))]` twin matching
existing increment functions.

- In `src/server/runtime.rs`, add a record helper near the existing
`record_supervisor_cancellation` logic. Have it emit `tracing::error!(event =
"server_supervisor_abnormal_termination", ...)` and call the new increment
function. Keep any label bounded and low cardinality; do not put the panic
message into a metric label.

Phase 2: Provide the awaitable shutdown handle

This phase adds the ServerShutdown handle, refactors the supervisor core so it can run as a spawned task, and wires the observer that retains the JoinHandle and records the terminal outcome. Group these tasks together, because they form one coherent runtime change.

Task 1: Refactor the supervisor core to accept an externally-owned `CancellationToken`

Separate the supervision logic from the token that stops it, so both the inline path and the new spawned path can reuse it.

• Update src/server/runtime.rs.
• Extract the body of run_with_shutdown (worker spawning, readiness signalling, await_supervisor_termination, tracker.close(), tracker.wait()) into a private helper that accepts the CancellationToken and the SupervisorLifecycle as parameters.
• Keep run_with_shutdown behaviour unchanged externally. It must still create the token locally, install the existing SupervisorCancellationDropGuard, and delegate to the helper.
• Keep run() unchanged.
• Preserve the graceful-drain semantics for in-flight connection tasks. Do not pass the token into spawn_connection_task.

Task 2: Introduce the `ServerShutdown` handle type

Provide a cloneable control handle over single-owner state, following the PushHandle R6 shape.

• Create the handle in a new module under src/server/, for example src/server/shutdown.rs, and re-export it from src/server/mod.rs.
• Wrap the shared state in a single Arc newtype. Derive Clone on the handle so a clone only bumps the refcount.
• Store a CancellationToken for stop() and a watch::Receiver of an Option of a Clone-able terminal descriptor for drained().
• Implement stop(&self): call token.cancel(). This is non-blocking and idempotent. Repeated calls converge to one cancellation.
• Implement async fn drained(&self) -> Result<(), ServerError>: clone the watch::Receiver, await until the value is Some, then map the terminal descriptor to Ok(()) or to the AbnormalTermination error. Several concurrent callers await the same channel and converge to one outcome.
• Define the terminal descriptor as a small Clone enum, for example Clean and Abnormal(String).

Task 3: Add the spawn method and the terminal observer

Start the supervisor as a spawned task, retain its JoinHandle, and record the terminal outcome so a panic is observed rather than swallowed.

• Update src/server/runtime.rs. Add a method on the Bound server, for example spawn(self) -> ServerShutdown.
• In the method: create the CancellationToken, spawn the supervisor helper future onto Tokio to obtain a JoinHandle<Result<(), ServerError>>, and create the watch channel for the terminal descriptor.
• Spawn one observer task that awaits the supervisor JoinHandle. On a clean return, send Clean. On a JoinError from a panic, capture the panic message, send Abnormal(message), and call the Task 1.3 record helper to emit the error-level event and metric.
• Keep the supervisor JoinHandle owned by the observer task, so nothing drops or aborts it.
• Return a ServerShutdown that holds the token clone and the watch::Receiver.
• Because the supervisor helper closes and drains the tracker before it returns, drained() resolving after supervisor completion guarantees that every accept loop has exited and the listener is released.

🤖 Prompt for AI agents

Implement Phase 2 of the awaitable server shutdown work: add the
`ServerShutdown` handle, refactor the supervisor core for reuse, and wire the
observer that retains the `JoinHandle`.

- In `src/server/runtime.rs`, extract the body of `run_with_shutdown` (worker
spawning, readiness signalling, `await_supervisor_termination`,
`tracker.close()`, `tracker.wait()`) into a private helper that accepts the
`CancellationToken` and the `SupervisorLifecycle` as parameters. Keep
`run_with_shutdown` behaviour unchanged externally: it must still create the
token locally, install the existing `SupervisorCancellationDropGuard`, and
delegate to the new helper. Keep `run()` unchanged. Preserve graceful-drain
semantics for in-flight connection tasks; do not pass the token into
`spawn_connection_task`.

- Create `src/server/shutdown.rs` and re-export `ServerShutdown` from
`src/server/mod.rs`. Wrap the shared state in a single `Arc` newtype and derive
`Clone` on the handle so cloning only bumps the refcount. Store a
`CancellationToken` for `stop()` and a `watch::Receiver` of an `Option` of a
`Clone`-able terminal descriptor for `drained()`. Define the terminal descriptor
as a small `Clone` enum with variants such as `Clean` and `Abnormal(String)`.
- Implement `stop(&self)` to call `token.cancel()`. This must be non-blocking
and idempotent; repeated calls converge to one cancellation.
- Implement `async fn drained(&self) -> Result<(), ServerError>` by cloning the
`watch::Receiver`, awaiting until the value is `Some`, then mapping the terminal
descriptor to `Ok(())` or to the `AbnormalTermination` error. Several concurrent
callers must converge to one outcome.

- In `src/server/runtime.rs`, add a `spawn(self) -> ServerShutdown` method on
the `Bound` server. Inside it: create the `CancellationToken`, spawn the
supervisor helper future onto Tokio to obtain a `JoinHandle<Result<(),
ServerError>>`, and create the `watch` channel for the terminal descriptor.
Spawn one observer task that awaits the supervisor `JoinHandle`: on clean
return, send `Clean`; on a `JoinError` from a panic, capture the panic message,
send `Abnormal(message)`, and call the Phase 1 record helper to emit the
error-level event and metric. Ensure the observer task owns the supervisor
`JoinHandle` so nothing drops or aborts it. Return a `ServerShutdown` holding
the token clone and the `watch::Receiver`. Rely on the supervisor helper closing
and draining the tracker before returning, so `drained()` resolving after
supervisor completion guarantees every accept loop has exited and the listener
is released.

Phase 3: Verify behaviour and update the developer documentation

This phase proves the acceptance criteria with tests that reuse the existing harness, and updates the developer guide to describe the new contract.

Task 1: Add lifecycle tests for the awaitable shutdown

Prove the drain guarantee, the graceful-drain preservation, the abnormal-termination path, convergence, and non-blocking delivery.

• Add tests to tests/server.rs or a new sibling test file, reusing unused_listener, wait_for_server_readiness, wait_for_listener_release, and factory from wireframe_testing.
• Test the drain: call spawn, stop(), drained().await, then assert the address refuses connections immediately, with no polling and no sleep beyond the existing wait_for_listener_release helper.
• Test graceful drain: open an in-flight connection, call stop(), and assert the existing graceful-drain semantics for in-flight connections are unchanged.
• Test abnormal termination: cause a panic inside the supervisor path, await drained(), assert the AbnormalTermination variant, and assert the error-level log event. Follow the observability test pattern in tests/server_lifecycle_observability.rs.
• Test convergence: issue several concurrent stop() calls and several concurrent drained() awaits, then assert one drain and one terminal outcome.
• Test non-blocking delivery: saturate the server, call stop(), and assert the call returns and the drain completes.

Task 2: Update the developer guide

Describe the new awaitable shutdown contract alongside the existing supervisor lifecycle section.

• Update docs/developers-guide.md, in and around the "Server supervisor lifecycle" section.
• Describe ServerShutdown, stop(), and drained(), and the guarantee that drained() resolves only after every accept loop has exited and the listener is released.
• Describe the new AbnormalTermination error variant, the error-level event server_supervisor_abnormal_termination, and the counter wireframe_server_supervisor_abnormal_terminations_total.
• State that run() behaviour and in-flight graceful drain are unchanged.
• Cross-reference ADR-014.

🤖 Prompt for AI agents

Implement Phase 3 of the awaitable server shutdown work: add lifecycle tests and
update the developer guide.

- Add tests to `tests/server.rs` or a new sibling test file, reusing the
`wireframe_testing` helpers `unused_listener`, `wait_for_server_readiness`,
`wait_for_listener_release`, and `factory`.
- Test the drain: call `spawn`, `stop()`, then `drained().await`. Assert the
address refuses connections immediately. Use no polling and no sleep beyond the
existing `wait_for_listener_release` helper.
- Test graceful drain: open an in-flight connection, call `stop()`, and assert
existing graceful-drain semantics for in-flight connections stay unchanged.
- Test abnormal termination: cause a panic inside the supervisor path, await
`drained()`, assert the result is the `AbnormalTermination` variant, and assert
the error-level log event fires. Follow the observability test pattern used in
`tests/server_lifecycle_observability.rs`.
- Test convergence: issue several concurrent `stop()` calls and several
concurrent `drained()` awaits, and assert exactly one drain and one terminal
outcome occur.
- Test non-blocking delivery: saturate the server, call `stop()`, and assert the
call returns immediately and the drain later completes.

- Update `docs/developers-guide.md` in and around the "Server supervisor
lifecycle" section. Describe `ServerShutdown`, `stop()`, and `drained()`,
including the guarantee that `drained()` resolves only after every accept loop
has exited and the listener is released. Describe the new `AbnormalTermination`
error variant, the error-level event `server_supervisor_abnormal_termination`,
and the counter `wireframe_server_supervisor_abnormal_terminations_total`. State
that `run()` behaviour and in-flight graceful drain stay unchanged.
Cross-reference ADR-014.

Research

WireframeServer lives in src/server/. The method run_with_shutdown in src/server/runtime.rs supervises the worker accept loops. It creates a CancellationToken and a TaskTracker as locals, spawns one accept loop per worker onto the tracker, fires the readiness signal, then races the caller-supplied shutdown future against tracker completion. It closes and drains the tracker before it returns Ok(()). A local SupervisorCancellationDropGuard (from #656) cancels the token if the supervisor future is dropped or aborted. run() is a thin wrapper that calls run_with_shutdown with a ctrl_c future.

ServerError in src/server/error.rs has two #[non_exhaustive] variants, Bind and Accept. Supervisor lifecycle metrics and tracing events live in src/metrics.rs and src/server/runtime.rs, keyed by the bounded ServerCancellationReason enum ("graceful", "dropped"). ADRs live in docs/, numbered adr-001 to adr-013. Lifecycle tests live in tests/server.rs, tests/server_lifecycle_model.rs, and tests/server_lifecycle_observability.rs, all built on wireframe_testing helpers (unused_listener, wait_for_server_readiness, wait_for_listener_release, factory).

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement Phase 1 of the awaitable server shutdown work: record the governing
ADR and add the typed error plus error-level observability for abnormal
termination.

- Create `docs/adr-014-server-shutdown-ownership.md`. Follow the section order
used by ADR 012 and ADR 013: Title, Status, Date, Context and problem statement,
Traceability, Decision drivers, Options considered (with a captioned comparison
table), Decision outcome (with captioned figures where useful), Consequences
(Positive/Negative), Rejected shortcuts, Migration plan, Verification,
Outstanding decisions, References. Set `## Status` to `Proposed` with a "First
proposed on <date> in issue `#N`" note. State that ADR-014 complements ADR 013 by
applying its sections 9 and 10 to the server runtime, and that it does not
supersede ADR 011, 012, or 013. Cite ADR 011 R5 and R6 as the ownership rules
for the new handle, and record the handle as a cloneable control over
single-owner state. Reuse ADR 013 vocabulary rather than inventing new terms.
Write in en-GB-oxendict spelling.

- In `src/server/error.rs`, add one new `#[non_exhaustive]`-compatible
`ServerError` variant, for example `AbnormalTermination`, carrying the captured
panic message as a `String`. Keep it distinct from `Bind`, `Accept`, and any
startup variants added by `#642`. Give it a clear `#[error(...)]` message that
includes the captured payload.

- In `src/metrics.rs`, add a counter constant, for example
`SERVER_SUPERVISOR_ABNORMAL_TERMINATIONS`, with value
`"wireframe_server_supervisor_abnormal_terminations_total"`, documented with the
`///` doc comment and `plaintext` HELP/TYPE block matching existing metrics. Add
`pub(crate) fn inc_server_supervisor_abnormal_termination(...)` with the paired
`#[cfg(feature = "metrics")]` / `#[cfg(not(feature = "metrics"))]` twin matching
existing increment functions.

- In `src/server/runtime.rs`, add a record helper near the existing
`record_supervisor_cancellation` logic. Have it emit `tracing::error!(event =
"server_supervisor_abnormal_termination", ...)` and call the new increment
function. Keep any label bounded and low cardinality; do not put the panic
message into a metric label.
===============================================================================

Task: 2

Implement Phase 2 of the awaitable server shutdown work: add the
`ServerShutdown` handle, refactor the supervisor core for reuse, and wire the
observer that retains the `JoinHandle`.

- In `src/server/runtime.rs`, extract the body of `run_with_shutdown` (worker
spawning, readiness signalling, `await_supervisor_termination`,
`tracker.close()`, `tracker.wait()`) into a private helper that accepts the
`CancellationToken` and the `SupervisorLifecycle` as parameters. Keep
`run_with_shutdown` behaviour unchanged externally: it must still create the
token locally, install the existing `SupervisorCancellationDropGuard`, and
delegate to the new helper. Keep `run()` unchanged. Preserve graceful-drain
semantics for in-flight connection tasks; do not pass the token into
`spawn_connection_task`.

- Create `src/server/shutdown.rs` and re-export `ServerShutdown` from
`src/server/mod.rs`. Wrap the shared state in a single `Arc` newtype and derive
`Clone` on the handle so cloning only bumps the refcount. Store a
`CancellationToken` for `stop()` and a `watch::Receiver` of an `Option` of a
`Clone`-able terminal descriptor for `drained()`. Define the terminal descriptor
as a small `Clone` enum with variants such as `Clean` and `Abnormal(String)`.
- Implement `stop(&self)` to call `token.cancel()`. This must be non-blocking
and idempotent; repeated calls converge to one cancellation.
- Implement `async fn drained(&self) -> Result<(), ServerError>` by cloning the
`watch::Receiver`, awaiting until the value is `Some`, then mapping the terminal
descriptor to `Ok(())` or to the `AbnormalTermination` error. Several concurrent
callers must converge to one outcome.

- In `src/server/runtime.rs`, add a `spawn(self) -> ServerShutdown` method on
the `Bound` server. Inside it: create the `CancellationToken`, spawn the
supervisor helper future onto Tokio to obtain a `JoinHandle<Result<(),
ServerError>>`, and create the `watch` channel for the terminal descriptor.
Spawn one observer task that awaits the supervisor `JoinHandle`: on clean
return, send `Clean`; on a `JoinError` from a panic, capture the panic message,
send `Abnormal(message)`, and call the Phase 1 record helper to emit the
error-level event and metric. Ensure the observer task owns the supervisor
`JoinHandle` so nothing drops or aborts it. Return a `ServerShutdown` holding
the token clone and the `watch::Receiver`. Rely on the supervisor helper closing
and draining the tracker before returning, so `drained()` resolving after
supervisor completion guarantees every accept loop has exited and the listener
is released.
===============================================================================

Task: 3

Implement Phase 3 of the awaitable server shutdown work: add lifecycle tests and
update the developer guide.

- Add tests to `tests/server.rs` or a new sibling test file, reusing the
`wireframe_testing` helpers `unused_listener`, `wait_for_server_readiness`,
`wait_for_listener_release`, and `factory`.
- Test the drain: call `spawn`, `stop()`, then `drained().await`. Assert the
address refuses connections immediately. Use no polling and no sleep beyond the
existing `wait_for_listener_release` helper.
- Test graceful drain: open an in-flight connection, call `stop()`, and assert
existing graceful-drain semantics for in-flight connections stay unchanged.
- Test abnormal termination: cause a panic inside the supervisor path, await
`drained()`, assert the result is the `AbnormalTermination` variant, and assert
the error-level log event fires. Follow the observability test pattern used in
`tests/server_lifecycle_observability.rs`.
- Test convergence: issue several concurrent `stop()` calls and several
concurrent `drained()` awaits, and assert exactly one drain and one terminal
outcome occur.
- Test non-blocking delivery: saturate the server, call `stop()`, and assert the
call returns immediately and the drain later completes.

- Update `docs/developers-guide.md` in and around the "Server supervisor
lifecycle" section. Describe `ServerShutdown`, `stop()`, and `drained()`,
including the guarantee that `drained()` resolves only after every accept loop
has exited and the listener is released. Describe the new `AbnormalTermination`
error variant, the error-level event `server_supervisor_abnormal_termination`,
and the counter `wireframe_server_supervisor_abnormal_terminations_total`. State
that `run()` behaviour and in-flight graceful drain stay unchanged.
Cross-reference ADR-014.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---