You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Gateway retries, timeouts and ejections are collected and then dropped #536
compose-inference-gateway configures a BackendTrafficPolicy (compose_failover_policy). It retries on 429, 503, reset and connect-failure, and it ejects an endpoint after five consecutive 5xx. compose-model-route sets a request timeout and streamIdleTimeout.
None of this can be seen. The collector's modelplane-gateway job scrapes Envoy's stats, but no mapping renames an envoy_* series, so filter/modelplane drops all of them. docs/content/models/model-service.md tells readers to watch envoy_cluster_upstream_rq_per_try_idle_timeout, which never reaches a sink.
When a canary resets streams or gets ejected, the only visible sign today is a change in latency.
All the endpoints behind one ModelService are one Envoy cluster, so these stats are per service, not per deployment. That is still the right level for "did adding this deployment raise retries".
envoy_http_downstream_rq_xx -> modelplane_gateway_requests_total{status_class}. This also covers 401, 404 and 429 responses that never reach a backend.
The mapping should also carry envoy_cluster_name onto service. Check the generated cluster name format first. Per-deployment detail comes from the usage records (see #535).
Some findings from testing these on the kind e2e (Envoy Gateway v1.8.4, AI Gateway v1.1.0, with compose_failover_policy's retries):
The fleet gateway's cluster is httproute/<mirrored-ns>/<service>/rule/0. Each cluster-gateway also reports httproute/<mirrored-ns>/<deployment>/rule/0, and the gateway job drops the gateway label, so the two aren't told apart without a filter like _in_flight_relabels.
per_try_idle_timeout counts attempts. A non-streamed request that stalled past idle counted 4, alongside 3 retries and one 504. timeout counts requests.
A stream cut short by idle reached the caller as a 200 and counted as 2xx in upstream_rq_xx. Only per_try_idle_timeout recorded it.
upstream_rq_xx counts once per request, by the final code, including Envoy's own 503s and 504s.
retry_limit_exceeded counted the idle-timeout failure but not a 503 that also used every retry.
I'd like to take this one. I've worked through your kind e2e findings, and the mapping below handles each of them.
map retries, ejections, timeouts, resets and upstream responses for the fleet gateway's httproute/<ns>/<service>/rule/0 clusters, with namespace and service, through _in_flight_relabels. It reads the gateway target label before clearing it, so the cluster-gateways' <deployment> clusters are dropped (collector.py:236-267). Listener response classes become modelplane_gateway_requests_total.
give modelplane_gateway_timeouts_total a kind label: request counts requests, per_try and per_try_idle count attempts, and the docs say so (metrics.py:70-82, telemetry.md:95)
describe upstream_responses_total as one count per request by final code, including the gateway's own 503 and 504 (telemetry.md:93)
leave retry_limit_exceeded unmapped, since it missed a 503 that used every retry (collector.py:75-85)
table tests run the generated relabels over fleet gateway, cluster-gateway, stats-listener and Istio series (test_collector.py:983-1131), and all 289 function tests pass
Activity
negz commentedon Nov 4, 2026
Some findings from testing these on the kind e2e (Envoy Gateway v1.8.4, AI Gateway v1.1.0, with
compose_failover_policy's retries):httproute/<mirrored-ns>/<service>/rule/0. Each cluster-gateway also reportshttproute/<mirrored-ns>/<deployment>/rule/0, and the gateway job drops thegatewaylabel, so the two aren't told apart without a filter like_in_flight_relabels.per_try_idle_timeoutcounts attempts. A non-streamed request that stalled pastidlecounted 4, alongside 3 retries and one 504.timeoutcounts requests.idlereached the caller as a 200 and counted as 2xx inupstream_rq_xx. Onlyper_try_idle_timeoutrecorded it.upstream_rq_xxcounts once per request, by the final code, including Envoy's own 503s and 504s.retry_limit_exceededcounted the idle-timeout failure but not a 503 that also used every retry.Stop documenting metrics and merges the collector doesn't produce
chethanuk commentedon Nov 16, 2026
I'd like to take this one. I've worked through your kind e2e findings, and the mapping below handles each of them.
httproute/<ns>/<service>/rule/0clusters, withnamespaceandservice, through_in_flight_relabels. It reads thegatewaytarget label before clearing it, so the cluster-gateways'<deployment>clusters are dropped (collector.py:236-267). Listener response classes becomemodelplane_gateway_requests_total.modelplane_gateway_timeouts_totalakindlabel:requestcounts requests,per_tryandper_try_idlecount attempts, and the docs say so (metrics.py:70-82,telemetry.md:95)idlestill reaches the caller as a 200 and counts as 2xx, so onlyper_try_idlerecords it (model-service.md:188-191). This replaces the Gateway retries, timeouts and ejections are collected and then dropped #536 pointer from Stop documenting metrics and merges the collector doesn't produce #639.upstream_responses_totalas one count per request by final code, including the gateway's own 503 and 504 (telemetry.md:93)retry_limit_exceededunmapped, since it missed a 503 that used every retry (collector.py:75-85)test_collector.py:983-1131), and all 289 function tests passI have a draft PR ready and will link it here.
chethanuk commentedon Nov 16, 2026
Draft PR is up: #676
Map the gateway's retries, timeouts, resets and ejections