From 3e596474aaa736417184fae90d415ef94c4b3d9d Mon Sep 17 00:00:00 2001 From: vthwang Date: Sat, 24 Oct 2026 23:40:28 -0700 Subject: [PATCH 1/3] docs: design UptimeRobot monitoring split Define a safe customer-facing status model and an internal alerting boundary. - document public and internal monitor contracts - capture token containment and migration steps - specify UptimeRobot groups, allowlists, and acceptance checks Signed-off-by: vthwang --- docs/uptimerobot-monitoring-design.md | 395 ++++++++++++++++++++++++++ 1 file changed, 395 insertions(+) create mode 100644 docs/uptimerobot-monitoring-design.md diff --git a/docs/uptimerobot-monitoring-design.md b/docs/uptimerobot-monitoring-design.md new file mode 100644 index 0000000..98625ee --- /dev/null +++ b/docs/uptimerobot-monitoring-design.md @@ -0,0 +1,395 @@ +# UptimeRobot monitoring and public status design + +Status: proposed + +Observed: 2026-10-24 + +Scope: production and staging VTA Farm monitors, `status.ic3.dev`, and the +token-gated monitor endpoints in `vtafarm-api` + +## 1. Decision + +Use two independent monitoring surfaces: + +1. **Customer status** answers only whether a customer can use VTA Farm. It + contains three production monitors: Farm control plane, VTA connectivity, + and shared platform connectivity. +2. **Operator alerts** report dependencies, unhealthy workloads, capacity, + Rancher, and staging. They notify the internal team and never appear on a + public status page. + +Public probes are deliberately safe to call without a credential and return no +infrastructure detail. Internal probes use an environment-scoped secret in the +standard `Authorization: Bearer ...` header. Query-string authentication is +removed. + +`status.ic3.dev` must select customer monitors explicitly. It must not select +the whole dashboard group, because adding an internal monitor to that group +would silently publish it. + +## 2. Current-state findings + +The UptimeRobot account currently has this VTA-related layout: + +| Dashboard group | Monitor | Public on `status.ic3.dev` | +| --- | --- | --- | +| `VTI DevOps` | `Prod Workloads` | No | +| `VTI DevOps` | `Staging Workloads` | No | +| `VTA Farm` | `Prod Capacity` | Yes | +| `VTA Farm` | `Prod Health` | Yes | +| `VTA Farm` | `Staging Capacity` | Yes | +| `VTA Farm` | `Staging Health` | Yes | +| `VTA Farm` | `VTA Farm Rancher` | Yes | + +`ic3 Status` selects the complete `VTA Farm` monitor group. Its **Show the +monitor URL** option is enabled. Consequently, each public monitor detail page +shows the complete target URL. + +The existing internal endpoints authenticate with a shared `?token=...` query +parameter. The `Prod Capacity` public detail page therefore exposed the shared +monitor token. Treat that token as compromised; hiding the URL later does not +make it secret again. The same token protects health, workload, and capacity +responses, all of which disclose operator-only state. + +There is also a second public page, `VTA Farm Status`, hosted at +`status.firstperson.dev`. It currently contains the same five `VTA Farm` +monitors. Its URL display is disabled, but it still needs to be included in the +migration so that the old internal monitors are not left public through another +page. + +The `Prod Workloads` monitor already demonstrates the intended internal model: +it is in `VTI DevOps`, not attached to a public page, and notifies internal +contacts. + +## 3. What belongs on the public page + +A public status signal should answer a user question, not expose an +implementation symptom. + +| Public component | User question | Public name | +| --- | --- | --- | +| Farm control plane | Can I reach VTA Farm and perform core management operations? | `VTA Farm` | +| VTA data plane | Can a VTA be reached through the public network path? | `VTA Connectivity` | +| Shared platform | Can customers reach the shared platform services used by default-connected VTAs? | `Platform Connectivity` | + +The following are operator signals and must not be public components: + +- pod restarts, pending pods, image-pull failures, or CrashLoopBackOff; +- node CPU, memory, Longhorn capacity, or metrics collection failures; +- database, Kubernetes API, or Vault detail; +- Rancher availability; +- any staging monitor; +- customer names, namespaces, pod names, node names, IP addresses, thresholds, + or raw upstream error messages. + +An internal alert can lead to a public incident when it is confirmed to affect +customers, but it should not automatically become a public component merely +because it is useful to operators. + +## 4. Proposed HTTP interface + +### 4.1 Public status endpoints + +Add three unauthenticated, read-only endpoints: + +```text +GET /api/v1/status/farm +GET /api/v1/status/vta +GET /api/v1/status/platform +``` + +All three use the same small response contract: + +```json +{ + "status": "operational", + "checked_at": "2026-10-24T23:00:00Z" +} +``` + +When a customer-important check fails, respond with HTTP `503` and only: + +```json +{ + "status": "unavailable", + "checked_at": "2026-10-24T23:00:00Z" +} +``` + +Contract rules: + +- `200` means operational; `503` means unavailable. Unexpected handler errors + also produce a generic `503`, with detail logged internally. +- Add `Cache-Control: no-store` and `Content-Type: application/json`. +- Do not accept target URLs, hostnames, thresholds, or component selectors from + the request. That avoids turning the probes into an SSRF or discovery API. +- Do not return component-level failures, deployment state, database state, or + exception text. +- Run downstream probes concurrently with a short per-probe timeout and an + overall budget below UptimeRobot's timeout. A three-second component timeout + and eight-second overall budget are suitable starting values. +- Use public HTTPS hostnames for downstream connectivity checks, not cluster + Services. The result must cover DNS, Cloudflare, TLS, Traefik, and the + application listener. +- Apply a rate limit generous enough for UptimeRobot regions and manual support + checks. Do not reuse the authenticated admin/user limits. + +#### `GET /api/v1/status/farm` + +The inbound request already proves that the public API hostname, DNS, TLS, +Cloudflare route, ingress, and API process are reachable. The handler should +also perform cheap readiness checks for the dependencies required by core Farm +management operations: + +- database reachable; +- Kubernetes API reachable; +- Vault reachable, initialized, and unsealed. + +It returns only the generic public contract. The existing internal dependency +endpoint remains the place to learn which dependency failed. + +#### `GET /api/v1/status/vta` + +Probe one dedicated production canary VTA through its public `/health` URL. +The canary must: + +- be owned by a system account, never a customer; +- use the same ingress, certificate, image family, Vault integration, and + storage path as an ordinary production VTA; +- have a stable hostname and lifecycle independent of load tests or the VTA + pool; +- be excluded from automatic cleanup; +- be identified by server configuration, for example + `STATUS_CANARY_VTA_NAME`, rather than a caller-supplied parameter. + +The endpoint fails when the configured canary is missing, not in `running` +state, its public `/health` request times out, or the response does not satisfy +the VTA health contract. A single broken customer VTA does not make this global +component fail; such failures belong to the internal workload alert and the +affected customer's incident handling. + +#### `GET /api/v1/status/platform` + +Resolve the one `DomainPlatform` session from the database, require it to be in +`running` state, and probe the shared stack through its public hostnames: + +| Component | Probe | +| --- | --- | +| Platform VTA | `GET https://vta./health` | +| Mediator | `GET https://mediator./mediator/v1/readyz` | +| DID hosting | `GET https://dids./health` | +| VTC | `GET https://vtc./health` | + +All required probes must pass. Internally log which component failed; publicly +return only `operational` or `unavailable`. + +If product semantics later permit the VTC to be unavailable without affecting +the default VTA connection path, split it into its own public component rather +than silently treating a failed required probe as healthy. + +### 4.2 Internal operator endpoints + +Keep the existing diagnostic functions, but make their audience explicit: + +```text +GET /api/v1/monitor/health +GET /api/v1/monitor/workloads +GET /api/v1/monitor/capacity +``` + +Authenticate them with: + +```http +Authorization: Bearer +``` + +Use UptimeRobot's built-in Bearer authentication where available; its custom +header feature is an equivalent fallback. UptimeRobot supports authentication +and custom request headers for HTTP/API monitors, so the secret does not need +to be placed in the URL. See the official +[API monitoring guide](https://help.uptimerobot.com/en/articles/13628553-uptimerobot-api-monitoring) +and [advanced settings guide](https://help.uptimerobot.com/en/articles/11360863-uptimerobot-advanced-settings-customize-your-monitor-configuration). + +Authentication rules: + +- Accept the `Bearer` scheme case-insensitively and compare the token in + constant time. +- Do not accept `token`, `api_key`, or any credential in the query string. +- Return `404` when monitoring is disabled and `401` for a missing or invalid + credential. +- Use separate random tokens for production and staging. A token grants only + read access to that environment's monitor endpoints. +- Store the token in the existing external Kubernetes Secret and in + UptimeRobot's authentication field. Never write it to Helm values, Git, + monitor names, metadata, examples, logs, or incident text. +- Redact `Authorization` and query strings from application, ingress, + Cloudflare, and support logs. +- Confirm that Cloudflare and Traefik forward the `Authorization` header before + removing query support. + +Moving the secret to a header prevents URL disclosure through status pages, +browser history, referrers, and ordinary URL logs. It does not make the already +published token safe; that token still must be rotated. + +## 5. UptimeRobot organization + +### 5.1 Dashboard groups + +Use groups for the operator dashboard, not as the public-page security +boundary: + +| Group | Monitors | Audience | +| --- | --- | --- | +| `VTA Farm` | the three production public status monitors | Customers and operators | +| `VTI DevOps` | production dependencies, workloads, capacity, Rancher | Internal production on-call | +| `VTI Staging` | staging dependencies, workloads, capacity, and staging surfaces | Internal development team | + +Recommended monitor names and targets: + +| Monitor | Target | Public page | Notifications | +| --- | --- | --- | --- | +| `VTA Farm` | `/api/v1/status/farm` | Yes | customer-impact on-call | +| `VTA Connectivity` | `/api/v1/status/vta` | Yes | customer-impact on-call | +| `Platform Connectivity` | `/api/v1/status/platform` | Yes | customer-impact on-call | +| `Prod Dependencies` | `/api/v1/monitor/health` + Bearer | No | internal production | +| `Prod Workloads` | `/api/v1/monitor/workloads` + Bearer | No | internal production | +| `Prod Capacity` | `/api/v1/monitor/capacity` + Bearer | No | internal production | +| `VTA Farm Rancher` | existing target | No | internal production | +| `Staging Dependencies` | staging `/monitor/health` + Bearer | No | internal staging | +| `Staging Workloads` | staging `/monitor/workloads` + Bearer | No | internal staging | +| `Staging Capacity` | staging `/monitor/capacity` + Bearer | No | internal staging | + +Add tags such as `audience:public`, `audience:internal`, `env:prod`, and +`env:staging` for filtering and audits. Tags are helpful metadata, but the +public page should still be an explicit allowlist of the three monitor IDs. + +### 5.2 `status.ic3.dev` + +Keep the existing Murmurations selection unchanged unless that product owner +wants a separate redesign. For VTA Farm: + +1. Remove the selected `VTA Farm` group from the status page. +2. Add exactly the three new public monitor IDs. +3. Disable **Show the monitor URL** as defense in depth. UptimeRobot supports + hiding monitored URLs; see its official + [status page overview](https://uptimerobot.com/status-page/). +4. Keep the details page, uptime charts, percentages, and outage history if + desired; their targets and responses are now safe for a public audience. +5. Do not include staging, capacity, workload, dependency-detail, or Rancher + monitors. + +Repeat the allowlist change on `status.firstperson.dev`, or retire that page if +it is no longer intentionally customer-facing. Do not leave the old five +monitors attached to either public page. + +### 5.3 Polling and notification defaults + +Start with: + +| Class | Interval | Failure confirmation | Timeout | Recipient | +| --- | --- | --- | --- | --- | +| Public customer signal | 1 minute | 2 consecutive failures | 15 seconds | production on-call | +| Prod dependencies/workloads | 1 minute | 2 consecutive failures | 15 seconds | internal production | +| Prod capacity | 5 minutes | 2 consecutive failures | 15 seconds | internal production | +| Staging | 5 minutes | 2 consecutive failures | 15 seconds | internal staging | + +Use at least North America and Europe for customer-facing checks if the plan +supports multiple regions. Enable recovery notifications. Put planned rollout +windows in UptimeRobot maintenance so expected restarts do not create false +incidents. + +Capacity is intentionally slower because a transient CPU spike is not an +outage. If storage urgency and CPU/memory noise require different escalation, +split the combined capacity endpoint into separate internal monitors later. + +## 6. Token containment and migration order + +### P0: contain the published credential + +1. Disable **Show the monitor URL** on `ic3 Status` immediately. +2. Detach `Prod Capacity`, `Prod Health`, both staging monitors, and Rancher + from every public status page. Audit both `status.ic3.dev` and + `status.firstperson.dev`. +3. Rotate `MONITOR_TOKEN`. The currently published value must never be reused. +4. Verify that requests with the old value fail. Do not paste either value into + tickets, chat, logs, or this document. + +Steps 1 and 2 reduce continued exposure; step 3 is what invalidates the leaked +credential. Search-engine or CDN cache removal is optional cleanup, not a +substitute for rotation. + +### Phase 1: move internal monitors to header authentication + +1. Implement Bearer-header support and tests while temporarily retaining the + old query path only for the cutover window. +2. Put the monitors in a maintenance window. +3. Edit each internal UptimeRobot monitor to use a query-free URL and the + Bearer authentication field. +4. Rotate the server and UptimeRobot value to a new environment-scoped token. +5. Confirm successful checks and recovery notifications. +6. Remove query-token support from code and OpenAPI, redeploy, and verify that + even a valid current token in `?token=` receives `401`. +7. End the maintenance window. + +The temporary compatibility window should be measured in minutes, not releases. + +### Phase 2: add customer-status probes + +1. Provision the dedicated canary VTA and protect it from cleanup. +2. Implement the three `/api/v1/status/*` endpoints with generic responses, + fixed targets, concurrency, and bounded timeouts. +3. Add handler, middleware, timeout, and response-redaction tests. +4. Create the three query-free UptimeRobot monitors. +5. Exercise one controlled failure for each monitor and confirm both down and + recovery behavior. +6. Replace the public page's group selection with the explicit three-monitor + allowlist. + +### Phase 3: reorganize internal monitors + +Move production diagnostics to `VTI DevOps`, staging diagnostics to +`VTI Staging`, and keep only the three customer signals in `VTA Farm`. Prefer +moving and renaming the existing monitors over deleting them so incident +history is retained. + +## 7. Acceptance checklist + +### Security + +- No public status page, monitor detail page, monitor URL, or redirect contains + a secret. +- The compromised token returns `401` on all internal monitor endpoints. +- The current token succeeds only in the `Authorization` header and fails in a + query parameter. +- Production and staging use different tokens. +- Public endpoint bodies contain no infrastructure identifiers or raw errors. +- Both public status pages have been audited. + +### Customer behavior + +- `status.ic3.dev` shows only `VTA Farm`, `VTA Connectivity`, and + `Platform Connectivity` for this product. +- A Farm dependency failure makes only the customer-relevant Farm component + unavailable and also triggers the detailed internal dependency alert. +- A canary VTA ingress or health failure changes `VTA Connectivity`. +- A required shared-stack failure changes `Platform Connectivity`. +- A staging failure, isolated pod restart, Rancher outage, or capacity warning + does not appear on the public page. + +### Operations + +- Public and internal monitors notify the intended contact groups. +- Controlled failures create an alert only after the configured confirmation + threshold and send a recovery notification. +- Maintenance windows suppress planned rollout noise. +- Monitor names, group membership, tags, URLs, intervals, regions, contacts, + and public-page membership are recorded in an operator-owned inventory. + +## 8. Follow-up boundary + +UptimeRobot remains appropriate for externally visible reachability and the +small set of aggregate HTTP checks above. Detailed pod, node, and storage +telemetry will eventually be easier to route and deduplicate in +Prometheus/Alertmanager or an equivalent Kubernetes-native stack. That is a +separate improvement; it is not required to remove internal signals from the +public page or to fix the token exposure. From 771fc9692b0f5bb9b3212d32d689fd65c6ebe631 Mon Sep 17 00:00:00 2001 From: vthwang Date: Sat, 24 Oct 2026 23:47:49 -0700 Subject: [PATCH 2/3] feat: support bearer auth for monitor endpoints Move the documented UptimeRobot contract away from query-string secrets while preserving a short cutover bridge. - validate monitor bearer headers in constant time - document the OpenAPI security scheme - cover auth failures and migration behavior Signed-off-by: vthwang --- internal/apidocs/openapi.yaml | 50 ++++++++++++++-------------- internal/config/config.go | 2 +- internal/handler/monitor.go | 23 +++++++++++-- internal/handler/monitor_test.go | 57 ++++++++++++++++++++++++++++++++ internal/router/router.go | 3 +- 5 files changed, 105 insertions(+), 30 deletions(-) diff --git a/internal/apidocs/openapi.yaml b/internal/apidocs/openapi.yaml index 2465625..493f96a 100644 --- a/internal/apidocs/openapi.yaml +++ b/internal/apidocs/openapi.yaml @@ -85,6 +85,11 @@ components: type: apiKey in: cookie name: vtafarm_admin + MonitorBearerAuth: + type: http + scheme: bearer + bearerFormat: opaque + description: Environment-scoped MONITOR_TOKEN for external uptime polling. schemas: TrustRegistryState: @@ -1028,22 +1033,18 @@ paths: # ── Monitor (external uptime poller) ──────────────────────────────────────── # Polled by UptimeRobot: healthy → 200, anything wrong → 503 with the same - # body shape. Gated by the MONITOR_TOKEN shared secret (?token=); the routes - # answer 404 until the token is configured. + # body shape. Gated by the MONITOR_TOKEN shared secret in the Authorization + # Bearer header; the routes answer 404 until the token is configured. /api/v1/monitor/health: get: - summary: "(token) Dependency health — DB, K8s API, Vault" + summary: "(monitor bearer) Dependency health — DB, K8s API, Vault" description: | Checks the database, the Kubernetes API server, and Vault (including its sealed state). `vault: disabled` (VAULT_ADDR unset) is not an alarm. tags: [System] - parameters: - - name: token - in: query - required: true - schema: { type: string } - description: MONITOR_TOKEN shared secret + security: + - MonitorBearerAuth: [] responses: "200": description: All dependencies healthy @@ -1062,23 +1063,21 @@ paths: "503": description: One or more dependencies unhealthy (same body shape, status=alarm) "401": - description: Missing or wrong token + description: Missing or wrong Bearer token + "404": + description: Monitoring is disabled because MONITOR_TOKEN is not configured /api/v1/monitor/workloads: get: - summary: "(token) Unhealthy pods in user + infra namespaces" + summary: "(monitor bearer) Unhealthy pods in user + infra namespaces" description: | Scans `{K8S_NAMESPACE_PREFIX}-*` plus MONITOR_EXTRA_NAMESPACES for pods that are crash-looping, failing image pulls, restarted within the last MONITOR_RESTART_WINDOW_MIN minutes, or stuck Pending / not-Ready past MONITOR_PENDING_GRACE_MIN minutes. Job-owned pods are ignored. tags: [System] - parameters: - - name: token - in: query - required: true - schema: { type: string } - description: MONITOR_TOKEN shared secret + security: + - MonitorBearerAuth: [] responses: "200": description: No unhealthy pods @@ -1092,23 +1091,22 @@ paths: "503": description: Unhealthy pods found — `pods` lists namespace/pod/reason "401": - description: Missing or wrong token + description: Missing or wrong Bearer token + "404": + description: Monitoring is disabled because MONITOR_TOKEN is not configured /api/v1/monitor/capacity: get: - summary: "(token) Node CPU/memory/storage usage vs alarm thresholds" + summary: "(monitor bearer) Node CPU/memory/storage usage vs alarm thresholds" description: | Alarms when any node's live CPU or memory usage, or any Longhorn node's disk usage, crosses its threshold — and also when metrics-server or Longhorn stats are unreadable ("can't see" is never "all green"). Thresholds default from MONITOR_*_PCT and can be overridden per poll. tags: [System] + security: + - MonitorBearerAuth: [] parameters: - - name: token - in: query - required: true - schema: { type: string } - description: MONITOR_TOKEN shared secret - name: cpu in: query schema: { type: integer, minimum: 1, maximum: 100 } @@ -1140,7 +1138,9 @@ paths: "503": description: Threshold breached or stats source unreadable — see `issues` "401": - description: Missing or wrong token + description: Missing or wrong Bearer token + "404": + description: Monitoring is disabled because MONITOR_TOKEN is not configured # ── Auth — Admin ──────────────────────────────────────────────────────────── diff --git a/internal/config/config.go b/internal/config/config.go index ee736b6..1a1c9b6 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -46,7 +46,7 @@ type Config struct { // MonitorConfig configures the token-gated /api/v1/monitor/* endpoints polled // by an external uptime service (UptimeRobot). Empty Token disables them. type MonitorConfig struct { - Token string // shared secret, passed as ?token= by the poller + Token string // shared secret, passed as an Authorization Bearer token by the poller CPUPercent int // per-node CPU used/allocatable alarm threshold MemPercent int // per-node memory used/allocatable alarm threshold StoragePercent int // per-node Longhorn disk usage alarm threshold diff --git a/internal/handler/monitor.go b/internal/handler/monitor.go index cd6c5bc..4ebf212 100644 --- a/internal/handler/monitor.go +++ b/internal/handler/monitor.go @@ -6,6 +6,7 @@ import ( "log" "net/http" "strconv" + "strings" "time" "github.com/gin-gonic/gin" @@ -45,20 +46,36 @@ func NewMonitorHandler(db *gorm.DB, k8sClient *k8s.Client, cfg MonitorConfig) *M // TokenRequired gates the monitor group. These endpoints expose internal // cluster state yet must be reachable without a login (the poller can't do -// passkeys), so a shared secret in the URL is the gate. No MONITOR_TOKEN -// configured → 404, indistinguishable from the routes not existing. +// passkeys), so MONITOR_TOKEN is supplied as an Authorization Bearer token. +// No MONITOR_TOKEN configured → 404, indistinguishable from the routes not +// existing. func (h *MonitorHandler) TokenRequired(c *gin.Context) { if h.cfg.Token == "" { c.AbortWithStatusJSON(http.StatusNotFound, gin.H{"error": "not found"}) return } - if subtle.ConstantTimeCompare([]byte(c.Query("token")), []byte(h.cfg.Token)) != 1 { + authorization := c.GetHeader("Authorization") + token, ok := monitorBearerToken(authorization) + if authorization == "" { + // Undocumented cutover bridge: existing pollers still send the exposed + // query token. Remove this fallback immediately after they move to Bearer. + token, ok = c.GetQuery("token") + } + if !ok || subtle.ConstantTimeCompare([]byte(token), []byte(h.cfg.Token)) != 1 { c.AbortWithStatusJSON(http.StatusUnauthorized, gin.H{"error": "unauthorized"}) return } c.Next() } +func monitorBearerToken(header string) (string, bool) { + parts := strings.Fields(header) + if len(parts) != 2 || !strings.EqualFold(parts[0], "Bearer") { + return "", false + } + return parts[1], true +} + // Health — GET /api/v1/monitor/health. Dependency checks: DB, K8s API, Vault // (including sealed state). Vault reports "disabled" (not an alarm) when // VAULT_ADDR is unset so local dev without Vault doesn't alarm forever. diff --git a/internal/handler/monitor_test.go b/internal/handler/monitor_test.go index 3fe5e6b..acb4eb4 100644 --- a/internal/handler/monitor_test.go +++ b/internal/handler/monitor_test.go @@ -1,11 +1,68 @@ package handler import ( + "net/http" + "net/http/httptest" "testing" + "github.com/gin-gonic/gin" "github.com/ic3software/vtafarm-api/internal/k8s" ) +func TestMonitorTokenRequired(t *testing.T) { + gin.SetMode(gin.TestMode) + + tests := []struct { + name string + configured string + authorize string + queryToken string + wantStatus int + wantReached bool + }{ + {name: "disabled", configured: "", authorize: "Bearer monitor-secret", wantStatus: http.StatusNotFound}, + {name: "valid bearer", configured: "monitor-secret", authorize: "Bearer monitor-secret", wantStatus: http.StatusNoContent, wantReached: true}, + {name: "case insensitive scheme", configured: "monitor-secret", authorize: "bearer monitor-secret", wantStatus: http.StatusNoContent, wantReached: true}, + {name: "missing header", configured: "monitor-secret", wantStatus: http.StatusUnauthorized}, + {name: "wrong token", configured: "monitor-secret", authorize: "Bearer wrong-secret", wantStatus: http.StatusUnauthorized}, + {name: "wrong scheme", configured: "monitor-secret", authorize: "Basic monitor-secret", wantStatus: http.StatusUnauthorized}, + {name: "malformed bearer", configured: "monitor-secret", authorize: "Bearer monitor-secret extra", wantStatus: http.StatusUnauthorized}, + {name: "legacy query token accepted during cutover", configured: "monitor-secret", queryToken: "monitor-secret", wantStatus: http.StatusNoContent, wantReached: true}, + {name: "invalid bearer does not fall back to query", configured: "monitor-secret", authorize: "Bearer wrong-secret", queryToken: "monitor-secret", wantStatus: http.StatusUnauthorized}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + h := NewMonitorHandler(nil, nil, MonitorConfig{Token: tt.configured}) + reached := false + router := gin.New() + router.GET("/monitor", h.TokenRequired, func(c *gin.Context) { + reached = true + c.Status(http.StatusNoContent) + }) + + req := httptest.NewRequest(http.MethodGet, "/monitor", nil) + if tt.authorize != "" { + req.Header.Set("Authorization", tt.authorize) + } + if tt.queryToken != "" { + query := req.URL.Query() + query.Set("token", tt.queryToken) + req.URL.RawQuery = query.Encode() + } + resp := httptest.NewRecorder() + router.ServeHTTP(resp, req) + + if resp.Code != tt.wantStatus { + t.Fatalf("status = %d, want %d; body = %s", resp.Code, tt.wantStatus, resp.Body.String()) + } + if reached != tt.wantReached { + t.Fatalf("downstream reached = %v, want %v", reached, tt.wantReached) + } + }) + } +} + func TestCapacityIssues(t *testing.T) { healthy := func() *k8s.ClusterStats { return &k8s.ClusterStats{ diff --git a/internal/router/router.go b/internal/router/router.go index a56d0d6..7107ff6 100644 --- a/internal/router/router.go +++ b/internal/router/router.go @@ -91,7 +91,8 @@ func Setup( v1 := r.Group("/api/v1") // Monitor endpoints — polled by an external uptime service (UptimeRobot): - // healthy → 200, anything wrong → 503. Gated by MONITOR_TOKEN (?token=) + // healthy → 200, anything wrong → 503. Gated by MONITOR_TOKEN in the + // Authorization Bearer header // since the poller can't log in; 404 until the token is configured. mh := handler.NewMonitorHandler(db, k8sClient, handler.MonitorConfig{ Token: cfg.Monitor.Token, From 2ce9c7736183bb2b79ecb1e77550a313019b3dbe Mon Sep 17 00:00:00 2001 From: vthwang Date: Sun, 25 Oct 2026 00:01:22 -0700 Subject: [PATCH 3/3] feat: add public service status probes Expose customer-safe availability signals for the Farm control plane, platform VTA, and shared platform services. - probe fixed public targets with bounded timeouts - return generic status responses without infrastructure details - apply a shared per-IP rate limit and document the policy Signed-off-by: vthwang --- docs/uptimerobot-monitoring-design.md | 79 +++++--- internal/apidocs/openapi.yaml | 92 ++++++++++ internal/handler/status.go | 250 ++++++++++++++++++++++++++ internal/handler/status_test.go | 131 ++++++++++++++ internal/router/router.go | 14 ++ 5 files changed, 537 insertions(+), 29 deletions(-) create mode 100644 internal/handler/status.go create mode 100644 internal/handler/status_test.go diff --git a/docs/uptimerobot-monitoring-design.md b/docs/uptimerobot-monitoring-design.md index 98625ee..a5e86b9 100644 --- a/docs/uptimerobot-monitoring-design.md +++ b/docs/uptimerobot-monitoring-design.md @@ -125,14 +125,46 @@ Contract rules: the request. That avoids turning the probes into an SSRF or discovery API. - Do not return component-level failures, deployment state, database state, or exception text. -- Run downstream probes concurrently with a short per-probe timeout and an - overall budget below UptimeRobot's timeout. A three-second component timeout - and eight-second overall budget are suitable starting values. +- Run downstream probes concurrently with a fixed three-second per-probe + timeout and an eight-second overall budget, below UptimeRobot's timeout. - Use public HTTPS hostnames for downstream connectivity checks, not cluster Services. The result must cover DNS, Cloudflare, TLS, Traefik, and the application listener. -- Apply a rate limit generous enough for UptimeRobot regions and manual support - checks. Do not reuse the authenticated admin/user limits. +- Share a limit of 30 requests per minute per client IP across the three public + status endpoints. This leaves room for UptimeRobot retries and manual support + checks without reusing the authenticated admin/user limits. + +#### Public status rate-limit policy + +The `/api/v1/status` route group applies one in-memory sliding-window limit to +all three public endpoints: + +```text +GET /api/v1/status/farm +GET /api/v1/status/vta +GET /api/v1/status/platform +``` + +The rules are: + +- one shared bucket per client IP, not 30 requests per endpoint; +- at most 30 requests in the preceding 60 seconds; +- every request reaching the route group counts, including a check that later + returns `503`; +- request 31 returns HTTP `429 Too Many Requests` without running database, + Kubernetes, Vault, VTA, or platform probes; +- the three normal one-minute UptimeRobot monitors consume approximately three + requests per minute from a checking source, leaving headroom for retries and + manual checks; +- the limiter is per API replica and held in memory. It protects dependencies + from one noisy source but is not a distributed DDoS control; Cloudflare edge + rate limiting remains the appropriate additional control for distributed + traffic. + +Client identity comes from Gin's `ClientIP()`, so production proxy trust and +forwarded-IP handling must preserve the real Cloudflare/visitor address rather +than collapsing every request to the ingress address or trusting arbitrary +forwarded headers. #### `GET /api/v1/status/farm` @@ -150,23 +182,14 @@ endpoint remains the place to learn which dependency failed. #### `GET /api/v1/status/vta` -Probe one dedicated production canary VTA through its public `/health` URL. -The canary must: - -- be owned by a system account, never a customer; -- use the same ingress, certificate, image family, Vault integration, and - storage path as an ordinary production VTA; -- have a stable hostname and lifecycle independent of load tests or the VTA - pool; -- be excluded from automatic cleanup; -- be identified by server configuration, for example - `STATUS_CANARY_VTA_NAME`, rather than a caller-supplied parameter. +Resolve the one `DomainPlatform` session, require it to be in `running` state, +and probe its VTA through the public `/health` URL. The fixed platform VTA uses +the same runtime and public ingress path as other VTAs without tying global +status to a customer-owned session or requiring a second synthetic VTA. -The endpoint fails when the configured canary is missing, not in `running` -state, its public `/health` request times out, or the response does not satisfy -the VTA health contract. A single broken customer VTA does not make this global -component fail; such failures belong to the internal workload alert and the -affected customer's incident handling. +A single broken customer VTA does not make this global component fail; such +failures belong to the internal workload alert and the affected customer's +incident handling. #### `GET /api/v1/status/platform` @@ -175,7 +198,6 @@ Resolve the one `DomainPlatform` session from the database, require it to be in | Component | Probe | | --- | --- | -| Platform VTA | `GET https://vta./health` | | Mediator | `GET https://mediator./mediator/v1/readyz` | | DID hosting | `GET https://dids./health` | | VTC | `GET https://vtc./health` | @@ -335,14 +357,13 @@ The temporary compatibility window should be measured in minutes, not releases. ### Phase 2: add customer-status probes -1. Provision the dedicated canary VTA and protect it from cleanup. -2. Implement the three `/api/v1/status/*` endpoints with generic responses, +1. Implement the three `/api/v1/status/*` endpoints with generic responses, fixed targets, concurrency, and bounded timeouts. -3. Add handler, middleware, timeout, and response-redaction tests. -4. Create the three query-free UptimeRobot monitors. -5. Exercise one controlled failure for each monitor and confirm both down and +2. Add handler, middleware, timeout, and response-redaction tests. +3. Create the three query-free UptimeRobot monitors. +4. Exercise one controlled failure for each monitor and confirm both down and recovery behavior. -6. Replace the public page's group selection with the explicit three-monitor +5. Replace the public page's group selection with the explicit three-monitor allowlist. ### Phase 3: reorganize internal monitors @@ -371,7 +392,7 @@ history is retained. `Platform Connectivity` for this product. - A Farm dependency failure makes only the customer-relevant Farm component unavailable and also triggers the detailed internal dependency alert. -- A canary VTA ingress or health failure changes `VTA Connectivity`. +- A platform VTA ingress or health failure changes `VTA Connectivity`. - A required shared-stack failure changes `Platform Connectivity`. - A staging failure, isolated pod restart, Rancher outage, or capacity warning does not appear on the public page. diff --git a/internal/apidocs/openapi.yaml b/internal/apidocs/openapi.yaml index 493f96a..4c2717c 100644 --- a/internal/apidocs/openapi.yaml +++ b/internal/apidocs/openapi.yaml @@ -92,6 +92,18 @@ components: description: Environment-scoped MONITOR_TOKEN for external uptime polling. schemas: + PublicServiceStatus: + type: object + additionalProperties: false + required: [status, checked_at] + properties: + status: + type: string + enum: [operational, unavailable] + checked_at: + type: string + format: date-time + TrustRegistryState: type: object required: [enabled, state, phase, image, did, error] @@ -1031,6 +1043,86 @@ paths: type: string example: ok + # ── Public customer status ───────────────────────────────────────────────── + + /api/v1/status/farm: + get: + summary: Public VTA Farm control-plane availability + description: | + Reports whether the public API and the dependencies required for core + Farm management operations are available. The response deliberately + contains no dependency or infrastructure detail. + tags: [System] + security: [] + responses: + "200": + description: VTA Farm is operational + content: + application/json: + schema: { $ref: "#/components/schemas/PublicServiceStatus" } + "503": + description: VTA Farm is unavailable; the body remains generic + content: + application/json: + schema: { $ref: "#/components/schemas/PublicServiceStatus" } + "429": + description: Shared public-status rate limit exceeded + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + + /api/v1/status/vta: + get: + summary: Public VTA connectivity + description: | + Checks the platform stack's VTA through its public HTTPS health + endpoint. The caller cannot select the target. + tags: [System] + security: [] + responses: + "200": + description: The VTA data plane is operational + content: + application/json: + schema: { $ref: "#/components/schemas/PublicServiceStatus" } + "503": + description: The platform VTA is unavailable + content: + application/json: + schema: { $ref: "#/components/schemas/PublicServiceStatus" } + "429": + description: Shared public-status rate limit exceeded + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + + /api/v1/status/platform: + get: + summary: Public shared-platform connectivity + description: | + Checks the platform stack's mediator, DID hosting, and VTC through + their public HTTPS health endpoints. VTA connectivity is reported by + `/api/v1/status/vta`; the response does not identify the failing + component. + tags: [System] + security: [] + responses: + "200": + description: All required shared-platform components are operational + content: + application/json: + schema: { $ref: "#/components/schemas/PublicServiceStatus" } + "503": + description: A required shared-platform component is unavailable + content: + application/json: + schema: { $ref: "#/components/schemas/PublicServiceStatus" } + "429": + description: Shared public-status rate limit exceeded + content: + application/json: + schema: { $ref: "#/components/schemas/Error" } + # ── Monitor (external uptime poller) ──────────────────────────────────────── # Polled by UptimeRobot: healthy → 200, anything wrong → 503 with the same # body shape. Gated by the MONITOR_TOKEN shared secret in the Authorization diff --git a/internal/handler/status.go b/internal/handler/status.go new file mode 100644 index 0000000..08f94c7 --- /dev/null +++ b/internal/handler/status.go @@ -0,0 +1,250 @@ +package handler + +import ( + "context" + "errors" + "fmt" + "io" + "log" + "net/http" + "net/url" + "strings" + "time" + + "github.com/gin-gonic/gin" + "golang.org/x/sync/errgroup" + "gorm.io/gorm" + + "github.com/ic3software/vtafarm-api/internal/k8s" + "github.com/ic3software/vtafarm-api/internal/model" + "github.com/ic3software/vtafarm-api/internal/vault" +) + +const ( + statusProbeTimeout = 3 * time.Second + statusOverallTimeout = 8 * time.Second +) + +type StatusConfig struct { + ClusterDomain string + VaultAddr string + VaultSkipVerify bool +} + +type publicStatusChecks interface { + Farm(context.Context) error + VTA(context.Context) error + Platform(context.Context) error +} + +// StatusHandler exposes customer-safe aggregate health. Detailed failures are +// logged for operators and never returned to the caller. +type StatusHandler struct { + checks publicStatusChecks + timeout time.Duration + now func() time.Time +} + +func NewStatusHandler(db *gorm.DB, k8sClient *k8s.Client, cfg StatusConfig) *StatusHandler { + client := &http.Client{ + Timeout: statusProbeTimeout, + CheckRedirect: func(*http.Request, []*http.Request) error { + return http.ErrUseLastResponse + }, + } + return &StatusHandler{ + checks: &livePublicStatusChecks{ + db: db, k8s: k8sClient, cfg: cfg, http: client, + }, + timeout: statusOverallTimeout, + now: time.Now, + } +} + +func (h *StatusHandler) Farm(c *gin.Context) { + h.respond(c, "farm", h.checks.Farm) +} + +func (h *StatusHandler) VTA(c *gin.Context) { + h.respond(c, "vta", h.checks.VTA) +} + +func (h *StatusHandler) Platform(c *gin.Context) { + h.respond(c, "platform", h.checks.Platform) +} + +func (h *StatusHandler) respond(c *gin.Context, component string, check func(context.Context) error) { + ctx, cancel := context.WithTimeout(c.Request.Context(), h.timeout) + defer cancel() + + status := "operational" + code := http.StatusOK + if err := check(ctx); err != nil { + log.Printf("public status %s unavailable: %v", component, err) + status, code = "unavailable", http.StatusServiceUnavailable + } + c.Header("Cache-Control", "no-store") + c.JSON(code, gin.H{ + "status": status, + "checked_at": h.now().UTC().Format(time.RFC3339), + }) +} + +type livePublicStatusChecks struct { + db *gorm.DB + k8s *k8s.Client + cfg StatusConfig + http *http.Client +} + +func (h *livePublicStatusChecks) Farm(ctx context.Context) error { + if h.db == nil { + return errors.New("database unavailable") + } + if h.k8s == nil { + return errors.New("kubernetes client unavailable") + } + if h.cfg.VaultAddr == "" { + return errors.New("vault not configured") + } + + group, groupCtx := errgroup.WithContext(ctx) + group.Go(func() error { + componentCtx, cancel := context.WithTimeout(groupCtx, statusProbeTimeout) + defer cancel() + sqlDB, err := h.db.DB() + if err != nil { + return fmt.Errorf("database handle: %w", err) + } + if err := sqlDB.PingContext(componentCtx); err != nil { + return fmt.Errorf("database ping: %w", err) + } + return nil + }) + group.Go(func() error { + componentCtx, cancel := context.WithTimeout(groupCtx, statusProbeTimeout) + defer cancel() + if err := h.k8s.Ping(componentCtx); err != nil { + return fmt.Errorf("kubernetes ping: %w", err) + } + return nil + }) + group.Go(func() error { + componentCtx, cancel := context.WithTimeout(groupCtx, statusProbeTimeout) + defer cancel() + status, err := vault.Health(componentCtx, h.cfg.VaultAddr, h.cfg.VaultSkipVerify) + if err != nil { + return fmt.Errorf("vault health: %w", err) + } + if status != "ok" { + return fmt.Errorf("vault state: %s", status) + } + return nil + }) + return group.Wait() +} + +func (h *livePublicStatusChecks) VTA(ctx context.Context) error { + stack, err := h.platformStack(ctx) + if err != nil { + return err + } + + target := strings.TrimRight(stack.PublicURL(), "/") + "/health" + if err := validateStatusProbeURL(target, h.cfg.ClusterDomain); err != nil { + return fmt.Errorf("platform VTA target: %w", err) + } + return h.probe(ctx, target) +} + +func (h *livePublicStatusChecks) Platform(ctx context.Context) error { + stack, err := h.platformStack(ctx) + if err != nil { + return err + } + + targets := map[string]string{ + "mediator": "https://" + stack.MediatorFQDN() + "/mediator/v1/readyz", + "dids": strings.TrimRight(stack.DidsURL(), "/") + "/health", + "vtc": "https://" + stack.VtcFQDN() + "/health", + } + for component, target := range targets { + if err := validateStatusProbeURL(target, h.cfg.ClusterDomain); err != nil { + return fmt.Errorf("%s target: %w", component, err) + } + } + + group, groupCtx := errgroup.WithContext(ctx) + for component, target := range targets { + component, target := component, target + group.Go(func() error { + if err := h.probe(groupCtx, target); err != nil { + return fmt.Errorf("%s: %w", component, err) + } + return nil + }) + } + return group.Wait() +} + +func (h *livePublicStatusChecks) platformStack(ctx context.Context) (*model.SetupSession, error) { + if h.db == nil { + return nil, errors.New("database unavailable") + } + + var sessions []model.SetupSession + if err := h.db.WithContext(ctx). + Where("domain_type = ? AND mode = ?", model.DomainPlatform, model.ModeFullStack). + Limit(2).Find(&sessions).Error; err != nil { + return nil, fmt.Errorf("load platform stack: %w", err) + } + if len(sessions) != 1 { + return nil, fmt.Errorf("platform stack count: %d", len(sessions)) + } + stack := &sessions[0] + if stack.Status != "running" { + return nil, fmt.Errorf("platform stack state: %s", stack.Status) + } + return stack, nil +} + +func (h *livePublicStatusChecks) probe(ctx context.Context, target string) error { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil) + if err != nil { + return fmt.Errorf("build request: %w", err) + } + req.Header.Set("Accept", "application/json") + req.Header.Set("User-Agent", "vtafarm-api-status/1") + resp, err := h.http.Do(req) + if err != nil { + return fmt.Errorf("request: %w", err) + } + defer resp.Body.Close() + _, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 4096)) + if resp.StatusCode < 200 || resp.StatusCode >= 300 { + return fmt.Errorf("HTTP %d", resp.StatusCode) + } + return nil +} + +func validateStatusProbeURL(rawURL, clusterDomain string) error { + parsed, err := url.Parse(rawURL) + if err != nil { + return fmt.Errorf("parse URL: %w", err) + } + domain := strings.ToLower(strings.TrimSuffix(clusterDomain, ".")) + host := strings.ToLower(strings.TrimSuffix(parsed.Hostname(), ".")) + if parsed.Scheme != "https" || parsed.User != nil || host == "" { + return errors.New("target must be an HTTPS URL without credentials") + } + if domain == "" || (host != domain && !strings.HasSuffix(host, "."+domain)) { + return errors.New("target is outside the configured cluster domain") + } + if port := parsed.Port(); port != "" && port != "443" { + return errors.New("target uses a non-HTTPS port") + } + if parsed.RawQuery != "" || parsed.Fragment != "" { + return errors.New("target must not contain a query or fragment") + } + return nil +} diff --git a/internal/handler/status_test.go b/internal/handler/status_test.go new file mode 100644 index 0000000..c05f8fb --- /dev/null +++ b/internal/handler/status_test.go @@ -0,0 +1,131 @@ +package handler + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/gin-gonic/gin" +) + +type fakePublicStatusChecks struct { + farmErr error + vtaErr error + platformErr error +} + +func (f *fakePublicStatusChecks) Farm(context.Context) error { return f.farmErr } +func (f *fakePublicStatusChecks) VTA(context.Context) error { return f.vtaErr } +func (f *fakePublicStatusChecks) Platform(context.Context) error { return f.platformErr } + +func TestStatusHandlerResponses(t *testing.T) { + gin.SetMode(gin.TestMode) + fixed := time.Date(2026, 10, 24, 23, 0, 0, 0, time.FixedZone("test", -7*60*60)) + + tests := []struct { + name string + path string + configure func(*fakePublicStatusChecks) + wantCode int + wantStatus string + }{ + {name: "farm operational", path: "/farm", configure: func(*fakePublicStatusChecks) {}, wantCode: http.StatusOK, wantStatus: "operational"}, + {name: "VTA unavailable", path: "/vta", configure: func(f *fakePublicStatusChecks) { f.vtaErr = errors.New("private failure detail") }, wantCode: http.StatusServiceUnavailable, wantStatus: "unavailable"}, + {name: "platform unavailable", path: "/platform", configure: func(f *fakePublicStatusChecks) { f.platformErr = errors.New("private failure detail") }, wantCode: http.StatusServiceUnavailable, wantStatus: "unavailable"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + checks := &fakePublicStatusChecks{} + tt.configure(checks) + h := &StatusHandler{checks: checks, timeout: time.Second, now: func() time.Time { return fixed }} + router := gin.New() + router.GET("/farm", h.Farm) + router.GET("/vta", h.VTA) + router.GET("/platform", h.Platform) + + resp := httptest.NewRecorder() + router.ServeHTTP(resp, httptest.NewRequest(http.MethodGet, tt.path, nil)) + if resp.Code != tt.wantCode { + t.Fatalf("status code = %d, want %d; body = %s", resp.Code, tt.wantCode, resp.Body.String()) + } + var body map[string]any + if err := json.Unmarshal(resp.Body.Bytes(), &body); err != nil { + t.Fatal(err) + } + if body["status"] != tt.wantStatus || body["checked_at"] != "2026-10-25T06:00:00Z" { + t.Fatalf("body = %#v", body) + } + if len(body) != 2 { + t.Fatalf("public response leaked extra fields: %#v", body) + } + if resp.Header().Get("Cache-Control") != "no-store" { + t.Fatalf("Cache-Control = %q", resp.Header().Get("Cache-Control")) + } + if strings.Contains(resp.Body.String(), "private failure detail") { + t.Fatal("public response leaked the internal error") + } + }) + } +} + +func TestStatusProbe(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/ok": + w.WriteHeader(http.StatusOK) + case "/redirect": + http.Redirect(w, r, "/ok", http.StatusFound) + default: + http.Error(w, "down", http.StatusServiceUnavailable) + } + })) + defer server.Close() + + checks := &livePublicStatusChecks{http: &http.Client{ + Timeout: time.Second, + CheckRedirect: func(*http.Request, []*http.Request) error { + return http.ErrUseLastResponse + }, + }} + if err := checks.probe(t.Context(), server.URL+"/ok"); err != nil { + t.Fatalf("healthy probe failed: %v", err) + } + if err := checks.probe(t.Context(), server.URL+"/down"); err == nil { + t.Fatal("503 probe succeeded") + } + if err := checks.probe(t.Context(), server.URL+"/redirect"); err == nil { + t.Fatal("redirect probe succeeded") + } +} + +func TestValidateStatusProbeURL(t *testing.T) { + tests := []struct { + name string + url string + domain string + valid bool + }{ + {name: "subdomain", url: "https://vta.firstperson.dev/health", domain: "firstperson.dev", valid: true}, + {name: "explicit HTTPS port", url: "https://vta.firstperson.dev:443/health", domain: "firstperson.dev", valid: true}, + {name: "wrong scheme", url: "http://vta.firstperson.dev/health", domain: "firstperson.dev"}, + {name: "lookalike domain", url: "https://vta.firstperson.dev.example.com/health", domain: "firstperson.dev"}, + {name: "credentials", url: "https://user@example.com/health", domain: "example.com"}, + {name: "query", url: "https://vta.firstperson.dev/health?target=other", domain: "firstperson.dev"}, + {name: "nonstandard port", url: "https://vta.firstperson.dev:8443/health", domain: "firstperson.dev"}, + {name: "missing cluster domain", url: "https://vta.firstperson.dev/health", domain: ""}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + err := validateStatusProbeURL(tt.url, tt.domain) + if (err == nil) != tt.valid { + t.Fatalf("error = %v, valid = %v", err, tt.valid) + } + }) + } +} diff --git a/internal/router/router.go b/internal/router/router.go index 7107ff6..30d4f07 100644 --- a/internal/router/router.go +++ b/internal/router/router.go @@ -89,6 +89,20 @@ func Setup( } v1 := r.Group("/api/v1") + statusHandler := handler.NewStatusHandler(db, k8sClient, handler.StatusConfig{ + ClusterDomain: cfg.ClusterDomain, + VaultAddr: cfg.Vault.Addr, + VaultSkipVerify: cfg.Vault.SkipVerify, + }) + // Three one-minute UptimeRobot checks consume about three requests per + // minute. Thirty leaves room for retries and manual checks without letting + // one source continuously fan out into dependency probes. + status := v1.Group("/status", middleware.NoStore(), middleware.RateLimit(30, time.Minute)) + { + status.GET("/farm", statusHandler.Farm) + status.GET("/vta", statusHandler.VTA) + status.GET("/platform", statusHandler.Platform) + } // Monitor endpoints — polled by an external uptime service (UptimeRobot): // healthy → 200, anything wrong → 503. Gated by MONITOR_TOKEN in the