Deep-Axe · Oct 26, 2026 · Oct 26, 2026
diff --git a/README.md b/README.md
@@ -153,6 +153,27 @@ Each answer lists the tools it called, and **Activity** shows the matching audit
 - **Access:** invite members, and issue or revoke API keys.
 - **Activity:** the audit log of every tool call, showing user, server, tool, time, duration and outcome.
 
+### Metrics
+
+Administrators see each server's calls, error rate and average and p95 latency for the last 24 hours on its **Connections** card. The same summary is available as JSON from `GET /api/metrics?hours=<1-168>`.
+
+The gateway serves Prometheus metrics on port `9464` at `/metrics`. The port is reachable inside the cluster only; the Ingress does not route it, and the pod carries `prometheus.io/scrape` annotations. To inspect it locally:
+
+```sh
+kubectl -n storelink port-forward svc/storelink-gateway 9464:9464
+curl -s localhost:9464/metrics
+```
+
+| Metric                                              | Type      | Labels                              |
+| --------------------------------------------------- | --------- | ----------------------------------- |
+| `storelink_tool_calls_total`                        | counter   | `team`, `server`, `slug`, `outcome` |
+| `storelink_tool_call_duration_seconds`              | histogram | `team`, `server`, `slug`            |
+| `storelink_upstream_up`                             | gauge     | `team`, `server`, `slug`            |
+| `storelink_upstream_status`                         | gauge     | `team`, `server`, `slug`, `status`  |
+| `storelink_upstream_last_failure_timestamp_seconds` | gauge     | `team`, `server`, `slug`            |
+
+For example, the error rate per server over five minutes is `sum by (server) (rate(storelink_tool_calls_total{outcome=~"failed|timeout|uncertain"}[5m])) / sum by (server) (rate(storelink_tool_calls_total{outcome!="denied"}[5m]))`.
+
 ## Configuration
 
 Common chart values:
@@ -165,6 +186,7 @@ Common chart values:
 | `chat.limits.perUserConcurrent` / `perTeamConcurrent` / `globalConcurrent` | `1` / `4` / `16`                                             | Concurrent chat turns                                    |
 | `gateway.extraUpstreamOrigins`                                             | `[]`                                                         | Cluster-internal origins allowed as upstreams            |
 | `gateway.healthIntervalMs`                                                 | `1000`                                                       | Interval between upstream health probes                  |
+| `gateway.metrics.enabled`, `gateway.metrics.port`                          | `true`, `9464`                                               | Prometheus metrics listener inside the cluster           |
 | `store.host`, `gateway.host`                                               | `store.storelink.localhost`, `dashboard.storelink.localhost` | Public hostnames                                         |
 | `store.timezone`                                                           | `Asia/Kolkata`                                               | Store timezone, which defines "today"                    |
 | `storage.className`                                                        | `local-path`                                                 | Storage class for the persistent volumes                 |
@@ -297,7 +319,7 @@ The smoke tests drive Google Chrome through Playwright. `npm run dev:dashboard`
 | #1 VPS deployment        | Documented and templated, not deployed |
 | #6 Security hardening    | Partial                                |
 | #7 Scaling               | Partial: rate and concurrency limits   |
-| #8 Observability         | Partial: health and audit history      |
+| #8 Observability         | Implemented                            |
 | #9 Upgrades and rollback | Documented and rehearsed               |
 | #2–#5, multi-store bonus | Not attempted                          |
 

diff --git a/apps/dashboard/src/main.tsx b/apps/dashboard/src/main.tsx
@@ -26,6 +26,15 @@ type Server = {
   lastSuccessAt?: string;
   lastFailureAt?: string;
   lastError?: string;
+  metrics?: {
+    calls: number;
+    errors: number;
+    timeouts: number;
+    denied: number;
+    errorRate: number;
+    avgLatencyMs: number;
+    p95LatencyMs: number;
+  };
 };
 type Member = {
   id: string;
@@ -441,6 +450,28 @@ function Connections({ me, run }: { me: Me; run: Run }) {
                   </dd>
                 </div>
               )}
+              {server.metrics && (
+                <>
+                  <div>
+                    <dt>Calls (24h)</dt>
+                    <dd>{server.metrics.calls}</dd>
+                  </div>
+                  <div>
+                    <dt>Error rate</dt>
+                    <dd>
+                      {server.metrics.errorRate}% · {server.metrics.errors}{' '}
+                      errors, {server.metrics.timeouts} timeouts
+                    </dd>
+                  </div>
+                  <div>
+                    <dt>Latency</dt>
+                    <dd>
+                      {server.metrics.avgLatencyMs} ms avg ·{' '}
+                      {server.metrics.p95LatencyMs} ms p95
+                    </dd>
+                  </div>
+                </>
+              )}
             </dl>
             {me.user.role === 'admin' && (
               <div className="actions">

diff --git a/charts/storelink/templates/apps.yaml b/charts/storelink/templates/apps.yaml
@@ -9,6 +9,9 @@ spec:
   template:
     metadata:
       labels: { storelink/component: gateway, app.kubernetes.io/instance: {{ .Release.Name }} }
+      {{- if .Values.gateway.metrics.enabled }}
+      annotations: { prometheus.io/scrape: "true", prometheus.io/port: {{ .Values.gateway.metrics.port | quote }}, prometheus.io/path: /metrics }
+      {{- end }}
     spec:
       serviceAccountName: {{ include "storelink.name" . }}-gateway
       automountServiceAccountToken: true
@@ -49,13 +52,20 @@ spec:
             - { name: CHAT_PER_TEAM_CONCURRENT, value: {{ .Values.chat.limits.perTeamConcurrent | quote }} }
             - { name: CHAT_GLOBAL_CONCURRENT, value: {{ .Values.chat.limits.globalConcurrent | quote }} }
             - { name: HEALTH_INTERVAL_MS, value: {{ .Values.gateway.healthIntervalMs | quote }} }
+            {{- if .Values.gateway.metrics.enabled }}
+            - { name: METRICS_PORT, value: {{ .Values.gateway.metrics.port | quote }} }
+            {{- end }}
             {{- if .Values.chat.enabled }}
             - { name: OPENAI_API_KEY_FILE, value: /secrets/openai-api-key }
             {{- end }}
             - { name: WOO_MCP_URL, value: "http://{{ include "storelink.name" . }}-woo-mcp:3001/mcp" }
             - { name: GATEWAY_ORIGIN, value: {{ .Values.gateway.publicUrl | quote }} }
             - { name: TEAM_ID, value: {{ .Values.teamId | quote }} }
-          ports: [{ name: http, containerPort: 3000 }]
+          ports:
+            - { name: http, containerPort: 3000 }
+            {{- if .Values.gateway.metrics.enabled }}
+            - { name: metrics, containerPort: {{ .Values.gateway.metrics.port }} }
+            {{- end }}
           startupProbe: { httpGet: { path: /health/live, port: http }, periodSeconds: 2, failureThreshold: 60 }
           readinessProbe: { httpGet: { path: /health/ready, port: http }, periodSeconds: 5, timeoutSeconds: 5 }
           livenessProbe: { httpGet: { path: /health/live, port: http }, periodSeconds: 10 }
@@ -88,7 +98,11 @@ metadata:
   name: {{ include "storelink.name" . }}-gateway
 spec:
   selector: { storelink/component: gateway, app.kubernetes.io/instance: {{ .Release.Name }} }
-  ports: [{ port: 3000, targetPort: http }]
+  ports:
+    - { name: http, port: 3000, targetPort: http }
+    {{- if .Values.gateway.metrics.enabled }}
+    - { name: metrics, port: {{ .Values.gateway.metrics.port }}, targetPort: metrics }
+    {{- end }}
 ---
 apiVersion: apps/v1
 kind: Deployment

diff --git a/charts/storelink/values.yaml b/charts/storelink/values.yaml
@@ -27,6 +27,9 @@ store:
 gateway:
   extraUpstreamOrigins: []
   healthIntervalMs: 1000
+  metrics:
+    enabled: true
+    port: 9464
   host: dashboard.storelink.localhost
   publicUrl: http://dashboard.storelink.localhost
   resources:

diff --git a/docs/system-design.md b/docs/system-design.md
@@ -41,6 +41,7 @@ The gateway (`services/gateway/src/gateway.ts`) uses `@modelcontextprotocol/sdk`
 - **Namespacing and collision handling:** Public tool names follow `<slug>__<alias>` (capped at 64 characters). Each upstream has a unique slug per team (up to 16 characters), and tool names with special characters or length over 46 characters receive a deterministic `<prefix>_<sha256-8>` alias. The gateway stores the `<teamId>:<slug>__<alias>` route map in memory and dispatches calls by looking up the target upstream and original tool name. Colliding upstream tool names such as `get_status` on both `time` and `ops` are exposed as `time__get_status` and `ops__get_status`.
 - **Timeouts and bounds:** Upstream TCP connections have a 2-second timeout (`undici.Agent`), tool discovery has a 3-second deadline (up to 5 pages / 200 tools), and tool calls have a 15-second deadline (256 KiB response cap). Arguments are validated against the upstream's JSON Schema with Ajv before dispatch.
 - **Health probing and failure isolation:** The gateway pings each connected upstream every second (`gateway.healthIntervalMs`, 1-second deadline) and refreshes full tool catalogs every 15 seconds. If a health probe, discovery or tool call fails with a transport or timeout error, only that upstream is marked `Unreachable`, its routes are removed from the active catalog, and the `Unreachable` state is held for a 2-second retry window before rediscovery. Upstream MCP client transports reset on probe or discovery failure so a restarted pod receives a fresh session. Status transitions and timestamps are persisted to PostgreSQL (`upstream_events` and `upstream_servers`).
+- **Metrics:** Call metrics are derived from `audit_calls`, which already records every call's server, outcome and duration (`services/gateway/src/metrics.ts`). One aggregation query, shared by PostgreSQL and the SQLite test store, counts outcomes and fills a fixed latency histogram (5 ms to 15 s buckets). The gateway serves the result in Prometheus format on a separate in-cluster port (`gateway.metrics.port`, default `9464`) that the Ingress does not route, so tool names and call volumes are not public. Administrators also see a 24-hour summary per server (calls, error rate, average and p95 latency, with p95 interpolated from the histogram buckets) in the dashboard and at `GET /api/metrics`. Members do not receive team-wide metrics, matching the audit log's visibility.
 - **Downstream sessions and catalog notifications:** `/mcp` uses stateful Streamable HTTP sessions (`services/gateway/src/mcp.ts`) with JSON POST responses and a GET SSE stream. When a team's tool catalog changes (server added, edited, disabled, removed, failed or recovered), the gateway sends `notifications/tools/list_changed` only to active sessions belonging to that team. Every request rechecks user and API key status; sessions expire after 15 minutes idle (capped at 8 per user / 256 globally).
 - **Mutations and auditing:** Write tools (`create_product`, `update_order_status`) are annotated with `readOnlyHint: false` and are never automatically retried. If a connection drops or times out after a write request is sent, the adapter and gateway return and audit the outcome as `uncertain` so callers check state before retrying. Every tool call writes a `started` audit record to PostgreSQL before dispatch and updates it with the duration and outcome (`success`, `failed`, `timeout`, `uncertain`, or `denied`). If the audit write fails, the tool call is aborted.