Skip to content

Gateway retries, timeouts and ejections are collected and then dropped #536

Open
chethanuk/modelplane
#5

Description

@dennis-upbound

compose-inference-gateway configures a BackendTrafficPolicy (compose_failover_policy). It retries on 429, 503, reset and connect-failure, and it ejects an endpoint after five consecutive 5xx. compose-model-route sets a request timeout and streamIdleTimeout.

None of this can be seen. The collector's modelplane-gateway job scrapes Envoy's stats, but no mapping renames an envoy_* series, so filter/modelplane drops all of them. docs/content/models/model-service.md tells readers to watch envoy_cluster_upstream_rq_per_try_idle_timeout, which never reaches a sink.

When a canary resets streams or gets ejected, the only visible sign today is a change in latency.

All the endpoints behind one ModelService are one Envoy cluster, so these stats are per service, not per deployment. That is still the right level for "did adding this deployment raise retries".

Suggested built-in modelplane-envoy mapping:

  • envoy_cluster_upstream_rq_retry -> modelplane_gateway_retries_total
  • envoy_cluster_upstream_rq_timeout, _per_try_timeout, _per_try_idle_timeout -> modelplane_gateway_timeouts_total{kind}
  • envoy_cluster_upstream_rq_rx_reset and envoy_cluster_upstream_cx_destroy_remote_with_active_rq -> modelplane_gateway_upstream_resets_total
  • envoy_cluster_outlier_detection_ejections_enforced_total -> modelplane_gateway_ejections_total
  • envoy_cluster_upstream_rq_xx -> modelplane_gateway_upstream_responses_total{status_class}
  • envoy_http_downstream_rq_xx -> modelplane_gateway_requests_total{status_class}. This also covers 401, 404 and 429 responses that never reach a backend.

The mapping should also carry envoy_cluster_name onto service. Check the generated cluster name format first. Per-deployment detail comes from the usage records (see #535).

Activity

  1. negz commented on Nov 4, 2026

    @negz
    Collaborator

    Some findings from testing these on the kind e2e (Envoy Gateway v1.8.4, AI Gateway v1.1.0, with compose_failover_policy's retries):

    • The fleet gateway's cluster is httproute/<mirrored-ns>/<service>/rule/0. Each cluster-gateway also reports httproute/<mirrored-ns>/<deployment>/rule/0, and the gateway job drops the gateway label, so the two aren't told apart without a filter like _in_flight_relabels.
    • per_try_idle_timeout counts attempts. A non-streamed request that stalled past idle counted 4, alongside 3 retries and one 504. timeout counts requests.
    • A stream cut short by idle reached the caller as a 200 and counted as 2xx in upstream_rq_xx. Only per_try_idle_timeout recorded it.
    • upstream_rq_xx counts once per request, by the final code, including Envoy's own 503s and 504s.
    • retry_limit_exceeded counted the idle-timeout failure but not a 503 that also used every retry.
  2. moved this from Backlog to In progress in Modelplane Roadmapon Nov 13, 2026
  3. chethanuk commented on Nov 16, 2026

    @chethanuk

    I'd like to take this one. I've worked through your kind e2e findings, and the mapping below handles each of them.

    • map retries, ejections, timeouts, resets and upstream responses for the fleet gateway's httproute/<ns>/<service>/rule/0 clusters, with namespace and service, through _in_flight_relabels. It reads the gateway target label before clearing it, so the cluster-gateways' <deployment> clusters are dropped (collector.py:236-267). Listener response classes become modelplane_gateway_requests_total.
    • give modelplane_gateway_timeouts_total a kind label: request counts requests, per_try and per_try_idle count attempts, and the docs say so (metrics.py:70-82, telemetry.md:95)
    • document that a stream cut by idle still reaches the caller as a 200 and counts as 2xx, so only per_try_idle records it (model-service.md:188-191). This replaces the Gateway retries, timeouts and ejections are collected and then dropped #536 pointer from Stop documenting metrics and merges the collector doesn't produce #639.
    • describe upstream_responses_total as one count per request by final code, including the gateway's own 503 and 504 (telemetry.md:93)
    • leave retry_limit_exceeded unmapped, since it missed a 503 that used every retry (collector.py:75-85)
    • table tests run the generated relabels over fleet gateway, cluster-gateway, stats-listener and Istio series (test_collector.py:983-1131), and all 289 function tests pass

    I have a draft PR ready and will link it here.

  4. chethanuk commented on Nov 16, 2026

    @chethanuk

    Draft PR is up: #676

  5. added a commit that references this issue on Nov 17, 2026
    a8388e6
  6. added
    g/gateway-and-routingMake model access independent of where models run, with routing operators can control and extend
    on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    g/gateway-and-routingMake model access independent of where models run, with routing operators can control and extend

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions