Skip to content

feat(rwi): dedicated webhook runtime, event metrics, push retry and configurable queue - #267

Open
ftong2010 wants to merge 1 commit into
restsend:mainfrom
ftong2010:rwi-dedicated-worker
Open

feat(rwi): dedicated webhook runtime, event metrics, push retry and configurable queue#267
ftong2010 wants to merge 1 commit into
restsend:mainfrom
ftong2010:rwi-dedicated-worker

Conversation

@ftong2010

Copy link
Copy Markdown

Motivation

The RWI webhook handler runs sequential, blocking-ish HTTP POSTs to the
router on the shared SIP runtime. A slow webhook consumer therefore
competes with SIP signalling, the HTTP route path and the CDR saver — and a
backpressured router can stall unrelated work. There was also no visibility
into the event pipeline (sent vs delivered vs dropped, queue depth), no retry
on transient push failures, a hardcoded queue capacity, and a gap in the
event stream: call_ringing was never emitted for calls whose trunk
replied with a 183 (early media), which is the majority of trunk calls.

What this PR adds

  1. Dedicated RWI webhook runtime — the webhook handler now runs on its
    own tokio runtime (rwi-webhook threads), so webhook egress and any
    slow-router backpressure never contend with the SIP runtime.
  2. Event pipeline metrics — Prometheus counters/gauges covering the
    webhook path end-to-end (see table below).
  3. Push retry with backoff — transient failures (transport error, 5xx,
    429) are retried with exponential backoff (200 ms base, doubling, hard
    cap 5); permanent 4xx are not retried.
  4. Configurable queue length — the gateway→handler broadcast channel
    capacity is now a config knob instead of a hardcoded constant.
  5. Opt-in queueing-latency histogram — measures how long an event waits
    in the queue (enqueued → handler dequeued). The HTTP push time is
    excluded on purpose: a slow router inflates push time, not queue wait.
  6. call_ringing fix — the event is now emitted on either provisional
    (183 with SDP or 180 without), exactly once per call.

Configuration

Key Default Description
[proxy] rwi_webhook_worker_threads 2 Dedicated tokio workers for the webhook push consumer
[proxy] rwi_webhook_channel_size 512 Queue length (broadcast channel capacity)
[rwi_webhook] retries 0 Max retries after a failed push (hard cap 5, 200 ms base backoff doubling)
[rwi_webhook] track_queue_latency false Record the queueing-wait histogram rwi_event_queue_latency_seconds
[rwi_webhook] timeout_ms 5000 Per-request HTTP timeout (each retry attempt is bounded by it)

Metrics

Metric Type Labels Description
rwi_event_enqueued_total Counter event_type Events pushed into the queue by gateway dispatch
rwi_events_pushed_total Counter event_type Events delivered with a 2xx response
rwi_events_push_failed_total Counter event_type Pushes that errored or returned non-2xx
rwi_events_push_retries_total Counter event_type Retry attempts after a failed push
rwi_events_dropped_total Counter - Events lost to queue lag (consumer fell behind)
rwi_event_queue_size Gauge - Configured queue capacity
rwi_event_queue_current Gauge - Events currently queued (sampled every 5 s on the dedicated runtime)
rwi_event_queue_latency_seconds Histogram event_type Queueing wait (enqueued → handler dequeued); opt-in via track_queue_latency

Verification

  • Startup builds the dedicated runtime: SIP workers=8 Media workers=12 RWI webhook workers=2, with 2 rwi-webhook threads visible in the
    process.
  • Smoke test (5 cps, ~100 calls): rwi_event_enqueued_total 209 ==
    rwi_events_pushed_total 209, rwi_event_queue_current 0 — no drops,
    no backlog.
  • Load test (50 cps, recording + CDR + RWI enabled): 20,065 enqueued ==
    20,065 pushed, queue stayed drained, rwi_events_dropped_total 0.
  • call_ringing now fires exactly once per call (600 calls → 600
    call_ringing enqueued/pushed), including 183-early-media calls that
    previously emitted nothing.
  • Queue config honored end-to-end: rwi_webhook_channel_size = 1234
    rwi_event_queue_size 1234.
  • cargo check --workspace --bins clean; release image builds.

Docs

docs/rwi_events_reference.md (zh) and docs/rwi_events_reference_en.md
updated with the new config keys and the metrics table;
docs/observability.md gained an "RWI Events" section in the metrics
reference.

@shenjinti

Copy link
Copy Markdown
Contributor
  1. The call_ringing fix premise doesn't match current main — rebase and re-check.
  2. Dedicated runtime — good

@ftong2010

ftong2010 commented Sep 6, 2026

Copy link
Copy Markdown
Author
  1. The call_ringing fix premise doesn't match current main — rebase and re-check.
  2. Dedicated runtime — good

ok, actually i rebased with main and removed call_ringing change, too. let me update PR

@ftong2010
ftong2010 force-pushed the rwi-dedicated-worker branch from 94b94bd to bc7b9e6 Compare September 7, 2026 10:26
@ftong2010 ftong2010 changed the title feat(rwi): dedicated webhook runtime, event metrics, push retry and configurable queue (+ call_ringing fix) feat(rwi): dedicated webhook runtime, event metrics, push retry and configurable queue Sep 7, 2026
…ry and queue

Rebased onto current main, which already carries a fixed-count retry
loop, the event-type-aware dedup key and full-body (never-truncated)
delivery logging — this keeps those and layers the feature set on top:

- dedicated tokio runtime for the RWI webhook handler (queue drain no
  longer competes with call/media workers), spawned via
  utils::rwi_webhook_spawn
- configurable push retry: [proxy.locator_webhook] retries (default 0 =
  single attempt, hard cap 5) with exponential backoff (200 ms base);
  retryable = transport error / 5xx / 429 — supersedes main's fixed
  WEBHOOK_RETRY_COUNT(3)/500 ms outer loop with the richer policy inside
  send_payload
- event pipeline metrics: rwi_event_queue_{size,current} gauges,
  rwi_events_{pushed,push_failed,retries}_total counters with an
  event_type label, and opt-in rwi_event_queue_latency_seconds histogram
  (gateway enqueue → handler dequeue, excludes the HTTP push;
  [proxy.locator_webhook] track_queue_latency)
- configurable broadcast queue length: [proxy]
  rwi_webhook_channel_size (default 512)
- docs: observability.md metric list + rwi_events_reference*.md config

Existing retry tests updated for the config-driven policy (retries:
Some(3) preserves the 3/4-request expectations); all 26 webhook tests
pass.
@ftong2010
ftong2010 force-pushed the rwi-dedicated-worker branch from bc7b9e6 to 5dba598 Compare September 7, 2026 10:43
@ftong2010

Copy link
Copy Markdown
Author

@shenjinti updated and rebased with latest main

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants