Repository navigation
Add per-server metrics and Prometheus endpoint for monitoring - #5
Merged
Merged
Conversation
Derive tool-call metrics from the audit log with one aggregation query shared by PostgreSQL and SQLite: outcome counts per upstream and a fixed latency histogram from 5 ms to 15 s. The gateway serves Prometheus metrics on a separate in-cluster port (gateway.metrics.port, default 9464) that the Ingress does not route: call counters by outcome, a latency histogram, and upstream up, status and last-failure gauges. The pod carries prometheus.io scrape annotations. Administrators see each server's calls, error rate and average and p95 latency for the last 24 hours on its Connections card and at GET /api/metrics?hours=<1-168>. Members do not receive team-wide metrics.
Store the slug on each upstream event, record a Removed event when a server is deleted, and backfill existing events from the server list or the audit log. The dashboard falls back to the stored slug instead of the raw server ID.
…restarts Denied calls (disabled server, revoked key) now record the server whose namespace was requested, so per-server metrics and the activity log count them. The registry seeds last success, failure and error from PostgreSQL when it registers an upstream, so the last-failure gauge survives a gateway restart.
Revoked-key and session-owner-mismatch denials recorded by the MCP handler now store the server whose namespace was requested, matching the denials recorded inside gateway.call().
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements stand-out item #8 Observability by adding per-server tool-call metrics, an in-cluster Prometheus
/metricsendpoint, dashboard metrics on each connection card, and server-slug retention in connection history.Changes
services/gateway/src/metrics.ts,services/gateway/src/main.ts,charts/storelink/):/metricson a dedicated in-cluster port (gateway.metrics.port, default9464) withprometheus.io/scrapepod annotations and no public Ingress route.storelink_tool_calls_total), a cumulative latency histogram from5 msto15 s(storelink_tool_call_duration_seconds), and upstream connectivity/status/last-failure gauges (storelink_upstream_up,storelink_upstream_status,storelink_upstream_last_failure_timestamp_seconds).lastSuccessAt,lastFailureAt, andlastErrorfrom PostgreSQL on gateway startup so failure timestamps survive restarts.services/gateway/src/dashboard-api.ts,services/gateway/src/registry.ts,apps/dashboard/src/main.tsx):audit_callsfor administrators on the Connections cards and viaGET /api/metrics?hours=<1-168>.tool_unavailable,key_revoked,session_owner_mismatch) to the requested server namespace so per-serverdeniedcounters and audit rows identify the target server.services/gateway/migrations/0005_event_slugs.sql,services/gateway/src/registry.ts,apps/dashboard/src/main.tsx):slugonupstream_events, records aRemovedevent when a server is deleted, and backfills existing rows so connection history keeps readable names after a server is removed.README.md,docs/system-design.md,docs/verification.md):gateway.metrics.enabled,gateway.metrics.port), and updates stand-out item#8 ObservabilitytoImplemented.Validation
npm run typechecknpm test(41 / 41 tests passing, includingtests/metrics.test.tsand denial server-attribution assertions intests/mcp-notifications.test.ts)npm run build:dashboardnpm run format:checkhelm lint charts/storelink -f charts/storelink/values-local.yaml&values-prod.yamlstorelinkk3d cluster):/metricson port9464inside the cluster exports counters, histogram buckets, and upstream gauges whilehttp://dashboard.storelink.localhost/metricsvia public Ingress returns404.GET /api/metricsandGET /api/serversreturn 24h call counts, error rates, and avg/p95 latency to admins and403to members.key_revokedandtool_unavailablecalls record the targetserver_idinaudit_callsand incrementoutcome="denied"in/metrics.node scripts/smoke-cluster.tsandnode scripts/smoke-dashboard.tspass against the deployed cluster.