Skip to content

Map the gateway's retries, timeouts, resets and ejections - #676

Open
chethanuk wants to merge 1 commit into
modelplaneai:mainfrom
chethanuk:oss/536-envoy-upstream-metrics
Open

chethanuk wants to merge 1 commit into
modelplaneai:mainfrom
chethanuk:oss/536-envoy-upstream-metrics

Conversation

@chethanuk

@chethanuk chethanuk commented Nov 16, 2026 •

Copy link
Copy Markdown

Description of your changes

Fixes #536.

The collector scrapes the gateway's Envoy retry, timeout, reset, ejection and response-class counters, but nothing renames them, so filter/modelplane drops them.

This keeps the fleet gateway's copies with namespace and service, the same way envoy_cluster_upstream_rq_active already is, and maps them to modelplane_gateway_retries_total, _ejections_total, _timeouts_total, _upstream_resets_total and _upstream_responses_total. The gateway listeners' downstream response classes become modelplane_gateway_requests_total, with a listener label so http and https don't collide.

How it handles each of negz's kind e2e findings:

  1. Cluster-gateways also report httproute/<ns>/<deployment>/rule/0. The new series go through _in_flight_relabels, which reads the gateway target label before it clears it and keeps only gateway=inference-gateway (collector.py:236-267). The stats listener and Istio sidecar copies are dropped too (collector.py:244, collector.py:442-443).
  2. per_try_idle_timeout counts attempts, timeout counts requests. Timeouts carry kind: request, per_try or per_try_idle, so they're never summed by accident, and the docs say which count attempts (metrics.py:70-82, telemetry.md:95).
  3. A stream cut by idle reaches the caller as a 200. The model-service docs now say only per_try_idle records it (model-service.md:188-191). This replaces the Gateway retries, timeouts and ejections are collected and then dropped #536 pointer that Stop documenting metrics and merges the collector doesn't produce #639 added there.
  4. upstream_rq_xx counts once per request by final code. upstream_responses_total is documented that way, including the gateway's own 503 and 504 (telemetry.md:93).
  5. retry_limit_exceeded missed a 503 that used every retry, so it isn't mapped (collector.py:75-85).

Resets fold two counters, so they also carry kind. stream_reset counts requests and connection_closed counts connections closed with requests in flight (telemetry.md:96). Status classes are 1xx to 5xx.

I added table tests that run the generated relabels over fleet gateway, cluster-gateway, stats-listener and Istio series and check which are kept and with what labels (test_collector.py:982-1233). All 289 function tests pass.

I have:

  • Read and followed Modelplane's contribution process.
  • Run nix flake check (or ./nix.sh flake check) and made sure it passes. No nix here; ran the function tests and ruff directly.
  • Added or updated tests covering any composition function changes.
  • Signed off every commit with git commit -s.

@github-actions

Copy link
Copy Markdown

Docs preview: https://modelplane-docs-pr-676.vercel.app (ready once the site's Content workflow finishes)

@negz

negz commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

/review

Thanks @chethanuk!

The collector scraped Envoy's upstream retry, timeout, reset, ejection
and response-class counters but filter/modelplane dropped them because
nothing renamed them. Keep the inference gateway's copies (the
cluster-gateways also report httproute/<ns>/<deployment>/rule/0, so
require gateway=inference-gateway) with namespace and service, and map
them to modelplane_gateway_{retries,timeouts,upstream_resets,ejections,
upstream_responses}_total. Also map the gateway listeners' downstream
response classes to modelplane_gateway_requests_total.

Timeouts and resets fold several Envoy counters onto one name, so they
carry a kind label to keep the series apart.

Fixes modelplaneai#536.

Signed-off-by: ChethanUK <chethanuk@outlook.com>
@chethanuk
chethanuk force-pushed the oss/536-envoy-upstream-metrics branch from ccf9a06 to a8388e6 Compare November 17, 2026 04:36
@chethanuk
chethanuk marked this pull request as ready for review November 17, 2026 04:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

Gateway retries, timeouts and ejections are collected and then dropped

2 participants