From 94f51be7d77162550b3a6e33d863ba6277eba128 Mon Sep 17 00:00:00 2001 From: vthwang Date: Sun, 25 Oct 2026 15:39:01 -0700 Subject: [PATCH 1/3] feat: consolidate public status monitoring Expose complete sanitized component results while reducing the public surface to two customer-facing monitors. - combine VTA and shared-service probes under the platform status - report concurrent infra and platform component outcomes - remove the standalone VTA route and update tests and documentation Signed-off-by: vthwang --- docs/uptimerobot-monitoring-design.md | 128 ++++++++----- internal/apidocs/openapi.yaml | 87 +++++---- internal/handler/status.go | 259 ++++++++++++++++++-------- internal/handler/status_test.go | 115 ++++++++++-- internal/router/router.go | 3 +- 5 files changed, 403 insertions(+), 189 deletions(-) diff --git a/docs/uptimerobot-monitoring-design.md b/docs/uptimerobot-monitoring-design.md index a5e86b9..2acce87 100644 --- a/docs/uptimerobot-monitoring-design.md +++ b/docs/uptimerobot-monitoring-design.md @@ -12,8 +12,8 @@ token-gated monitor endpoints in `vtafarm-api` Use two independent monitoring surfaces: 1. **Customer status** answers only whether a customer can use VTA Farm. It - contains three production monitors: Farm control plane, VTA connectivity, - and shared platform connectivity. + contains two production monitors: VTA Farm infrastructure and the combined + VTA Farm platform. 2. **Operator alerts** report dependencies, unhealthy workloads, capacity, Rancher, and staging. They notify the internal team and never appear on a public status page. @@ -68,9 +68,8 @@ implementation symptom. | Public component | User question | Public name | | --- | --- | --- | -| Farm control plane | Can I reach VTA Farm and perform core management operations? | `VTA Farm` | -| VTA data plane | Can a VTA be reached through the public network path? | `VTA Connectivity` | -| Shared platform | Can customers reach the shared platform services used by default-connected VTAs? | `Platform Connectivity` | +| Farm infrastructure | Can I reach VTA Farm and perform core management operations? | `VTA Farm Infra` | +| Platform | Can customers reach the platform VTA and all shared services required by it? | `VTA Farm Platform` | The following are operator signals and must not be public components: @@ -90,32 +89,60 @@ because it is useful to operators. ### 4.1 Public status endpoints -Add three unauthenticated, read-only endpoints: +Add two unauthenticated, read-only endpoints: ```text GET /api/v1/status/farm -GET /api/v1/status/vta GET /api/v1/status/platform ``` -All three use the same small response contract: +The Infra endpoint reports one sanitized result for every required component: ```json { "status": "operational", - "checked_at": "2026-10-24T23:00:00Z" + "checked_at": "2026-10-24T23:00:00Z", + "components": { + "database": "operational", + "kubernetes": "operational", + "vault": "operational" + } } ``` -When a customer-important check fails, respond with HTTP `503` and only: +When multiple Infra checks fail, return every result with HTTP `503`: ```json { "status": "unavailable", - "checked_at": "2026-10-24T23:00:00Z" + "checked_at": "2026-10-24T23:00:00Z", + "components": { + "database": "unavailable", + "kubernetes": "operational", + "vault": "unavailable" + } } ``` +The Platform endpoint adds one sanitized result for every required component: + +```json +{ + "status": "unavailable", + "checked_at": "2026-10-24T23:00:00Z", + "components": { + "vta": "unavailable", + "mediator": "unavailable", + "dids": "operational", + "vtc": "operational" + } +} +``` + +All component values are `operational`, `unavailable`, or `unknown`. `unknown` +means a shared prerequisite, such as resolving the DomainPlatform session, +failed before the individual probes could run. + Contract rules: - `200` means operational; `503` means unavailable. Unexpected handler errors @@ -123,25 +150,25 @@ Contract rules: - Add `Cache-Control: no-store` and `Content-Type: application/json`. - Do not accept target URLs, hostnames, thresholds, or component selectors from the request. That avoids turning the probes into an SSRF or discovery API. -- Do not return component-level failures, deployment state, database state, or - exception text. +- Return only fixed Infra or Platform component names and sanitized states. Do + not return target URLs, deployment state, connection strings, or exception + text. - Run downstream probes concurrently with a fixed three-second per-probe timeout and an eight-second overall budget, below UptimeRobot's timeout. - Use public HTTPS hostnames for downstream connectivity checks, not cluster Services. The result must cover DNS, Cloudflare, TLS, Traefik, and the application listener. -- Share a limit of 30 requests per minute per client IP across the three public +- Share a limit of 30 requests per minute per client IP across the two public status endpoints. This leaves room for UptimeRobot retries and manual support checks without reusing the authenticated admin/user limits. #### Public status rate-limit policy The `/api/v1/status` route group applies one in-memory sliding-window limit to -all three public endpoints: +both public endpoints: ```text GET /api/v1/status/farm -GET /api/v1/status/vta GET /api/v1/status/platform ``` @@ -153,7 +180,7 @@ The rules are: returns `503`; - request 31 returns HTTP `429 Too Many Requests` without running database, Kubernetes, Vault, VTA, or platform probes; -- the three normal one-minute UptimeRobot monitors consume approximately three +- the two normal one-minute UptimeRobot monitors consume approximately two requests per minute from a checking source, leaving headroom for retries and manual checks; - the limiter is per API replica and held in memory. It protects dependencies @@ -170,40 +197,39 @@ forwarded headers. The inbound request already proves that the public API hostname, DNS, TLS, Cloudflare route, ingress, and API process are reachable. The handler should -also perform cheap readiness checks for the dependencies required by core Farm -management operations: +also concurrently perform cheap readiness checks for the dependencies required +by core Farm management operations: - database reachable; - Kubernetes API reachable; - Vault reachable, initialized, and unsealed. -It returns only the generic public contract. The existing internal dependency -endpoint remains the place to learn which dependency failed. - -#### `GET /api/v1/status/vta` - -Resolve the one `DomainPlatform` session, require it to be in `running` state, -and probe its VTA through the public `/health` URL. The fixed platform VTA uses -the same runtime and public ingress path as other VTAs without tying global -status to a customer-owned session or requiring a second synthetic VTA. +Allow every probe to finish even when another fails, and return the fixed +`database`, `kubernetes`, and `vault` states. Raw errors remain internal. -A single broken customer VTA does not make this global component fail; such -failures belong to the internal workload alert and the affected customer's -incident handling. +Do not add a `cloudflare` component to the response. If DNS, Cloudflare, TLS, +ingress, or the API process is unavailable, this endpoint cannot return any +JSON at all; UptimeRobot's ability to reach the endpoint is therefore the edge +status signal. A returned `cloudflare` field could only say `operational` and +would be misleading. #### `GET /api/v1/status/platform` Resolve the one `DomainPlatform` session from the database, require it to be in -`running` state, and probe the shared stack through its public hostnames: +`running` state, and probe the platform VTA and shared stack through their +public hostnames: | Component | Probe | | --- | --- | +| VTA | `GET /health` | | Mediator | `GET https://mediator./mediator/v1/readyz` | | DID hosting | `GET https://dids./health` | | VTC | `GET https://vtc./health` | -All required probes must pass. Internally log which component failed; publicly -return only `operational` or `unavailable`. +Launch all four probes concurrently and allow every probe to finish even when +another fails. All required probes must pass. Internally log the aggregated +errors; publicly return only the fixed component names and sanitized states. +A single broken customer-owned VTA does not make this global component fail. If product semantics later permit the VTC to be unavailable without affecting the default VTA connection path, split it into its own public component rather @@ -262,7 +288,7 @@ boundary: | Group | Monitors | Audience | | --- | --- | --- | -| `VTA Farm` | the three production public status monitors | Customers and operators | +| `VTA Farm` | the two production public status monitors | Customers and operators | | `VTI DevOps` | production dependencies, workloads, capacity, Rancher | Internal production on-call | | `VTI Staging` | staging dependencies, workloads, capacity, and staging surfaces | Internal development team | @@ -270,9 +296,8 @@ Recommended monitor names and targets: | Monitor | Target | Public page | Notifications | | --- | --- | --- | --- | -| `VTA Farm` | `/api/v1/status/farm` | Yes | customer-impact on-call | -| `VTA Connectivity` | `/api/v1/status/vta` | Yes | customer-impact on-call | -| `Platform Connectivity` | `/api/v1/status/platform` | Yes | customer-impact on-call | +| `VTA Farm Infra` | `/api/v1/status/farm` | Yes | customer-impact on-call | +| `VTA Farm Platform` | `/api/v1/status/platform` | Yes | customer-impact on-call | | `Prod Dependencies` | `/api/v1/monitor/health` + Bearer | No | internal production | | `Prod Workloads` | `/api/v1/monitor/workloads` + Bearer | No | internal production | | `Prod Capacity` | `/api/v1/monitor/capacity` + Bearer | No | internal production | @@ -283,7 +308,7 @@ Recommended monitor names and targets: Add tags such as `audience:public`, `audience:internal`, `env:prod`, and `env:staging` for filtering and audits. Tags are helpful metadata, but the -public page should still be an explicit allowlist of the three monitor IDs. +public page should still be an explicit allowlist of the two monitor IDs. ### 5.2 `status.ic3.dev` @@ -291,7 +316,7 @@ Keep the existing Murmurations selection unchanged unless that product owner wants a separate redesign. For VTA Farm: 1. Remove the selected `VTA Farm` group from the status page. -2. Add exactly the three new public monitor IDs. +2. Add exactly the two public monitor IDs. 3. Disable **Show the monitor URL** as defense in depth. UptimeRobot supports hiding monitored URLs; see its official [status page overview](https://uptimerobot.com/status-page/). @@ -357,19 +382,19 @@ The temporary compatibility window should be measured in minutes, not releases. ### Phase 2: add customer-status probes -1. Implement the three `/api/v1/status/*` endpoints with generic responses, - fixed targets, concurrency, and bounded timeouts. +1. Implement the two `/api/v1/status/*` endpoints with fixed targets, + sanitized responses, concurrency, and bounded timeouts. 2. Add handler, middleware, timeout, and response-redaction tests. -3. Create the three query-free UptimeRobot monitors. +3. Create the two query-free UptimeRobot monitors. 4. Exercise one controlled failure for each monitor and confirm both down and recovery behavior. -5. Replace the public page's group selection with the explicit three-monitor +5. Replace the public page's group selection with the explicit two-monitor allowlist. ### Phase 3: reorganize internal monitors Move production diagnostics to `VTI DevOps`, staging diagnostics to -`VTI Staging`, and keep only the three customer signals in `VTA Farm`. Prefer +`VTI Staging`, and keep only the two customer signals in `VTA Farm`. Prefer moving and renaming the existing monitors over deleting them so incident history is retained. @@ -383,17 +408,18 @@ history is retained. - The current token succeeds only in the `Authorization` header and fails in a query parameter. - Production and staging use different tokens. -- Public endpoint bodies contain no infrastructure identifiers or raw errors. +- Public endpoint bodies contain only the documented fixed component names and + sanitized states, with no targets, identifiers, or raw errors. - Both public status pages have been audited. ### Customer behavior -- `status.ic3.dev` shows only `VTA Farm`, `VTA Connectivity`, and - `Platform Connectivity` for this product. -- A Farm dependency failure makes only the customer-relevant Farm component +- `status.ic3.dev` shows only `VTA Farm Infra` and `VTA Farm Platform` for this + product. +- A Farm dependency failure makes only `VTA Farm Infra` unavailable and also triggers the detailed internal dependency alert. -- A platform VTA ingress or health failure changes `VTA Connectivity`. -- A required shared-stack failure changes `Platform Connectivity`. +- A platform VTA ingress, health, or required shared-stack failure changes + `VTA Farm Platform`; its endpoint reports every sanitized component state. - A staging failure, isolated pod restart, Rancher outage, or capacity warning does not appear on the public page. diff --git a/internal/apidocs/openapi.yaml b/internal/apidocs/openapi.yaml index 4c2717c..1a4ce07 100644 --- a/internal/apidocs/openapi.yaml +++ b/internal/apidocs/openapi.yaml @@ -92,10 +92,10 @@ components: description: Environment-scoped MONITOR_TOKEN for external uptime polling. schemas: - PublicServiceStatus: + PublicInfraStatus: type: object additionalProperties: false - required: [status, checked_at] + required: [status, checked_at, components] properties: status: type: string @@ -103,6 +103,41 @@ components: checked_at: type: string format: date-time + components: + $ref: "#/components/schemas/PublicInfraComponents" + + PublicInfraComponents: + type: object + additionalProperties: false + required: [database, kubernetes, vault] + properties: + database: { type: string, enum: [operational, unavailable, unknown] } + kubernetes: { type: string, enum: [operational, unavailable, unknown] } + vault: { type: string, enum: [operational, unavailable, unknown] } + + PublicPlatformStatus: + type: object + additionalProperties: false + required: [status, checked_at, components] + properties: + status: + type: string + enum: [operational, unavailable] + checked_at: + type: string + format: date-time + components: + $ref: "#/components/schemas/PublicPlatformComponents" + + PublicPlatformComponents: + type: object + additionalProperties: false + required: [vta, mediator, dids, vtc] + properties: + vta: { type: string, enum: [operational, unavailable, unknown] } + mediator: { type: string, enum: [operational, unavailable, unknown] } + dids: { type: string, enum: [operational, unavailable, unknown] } + vtc: { type: string, enum: [operational, unavailable, unknown] } TrustRegistryState: type: object @@ -1059,37 +1094,12 @@ paths: description: VTA Farm is operational content: application/json: - schema: { $ref: "#/components/schemas/PublicServiceStatus" } - "503": - description: VTA Farm is unavailable; the body remains generic - content: - application/json: - schema: { $ref: "#/components/schemas/PublicServiceStatus" } - "429": - description: Shared public-status rate limit exceeded - content: - application/json: - schema: { $ref: "#/components/schemas/Error" } - - /api/v1/status/vta: - get: - summary: Public VTA connectivity - description: | - Checks the platform stack's VTA through its public HTTPS health - endpoint. The caller cannot select the target. - tags: [System] - security: [] - responses: - "200": - description: The VTA data plane is operational - content: - application/json: - schema: { $ref: "#/components/schemas/PublicServiceStatus" } + schema: { $ref: "#/components/schemas/PublicInfraStatus" } "503": - description: The platform VTA is unavailable + description: One or more required infrastructure components are unavailable content: application/json: - schema: { $ref: "#/components/schemas/PublicServiceStatus" } + schema: { $ref: "#/components/schemas/PublicInfraStatus" } "429": description: Shared public-status rate limit exceeded content: @@ -1098,25 +1108,24 @@ paths: /api/v1/status/platform: get: - summary: Public shared-platform connectivity + summary: Public VTA Farm Platform availability description: | - Checks the platform stack's mediator, DID hosting, and VTC through - their public HTTPS health endpoints. VTA connectivity is reported by - `/api/v1/status/vta`; the response does not identify the failing - component. + Concurrently checks the platform VTA, mediator, DID hosting, and VTC + through their public HTTPS health endpoints. The component map reports + every sanitized result without exposing targets or raw errors. tags: [System] security: [] responses: "200": - description: All required shared-platform components are operational + description: All required VTA Farm Platform components are operational content: application/json: - schema: { $ref: "#/components/schemas/PublicServiceStatus" } + schema: { $ref: "#/components/schemas/PublicPlatformStatus" } "503": - description: A required shared-platform component is unavailable + description: One or more required components are unavailable or could not be checked content: application/json: - schema: { $ref: "#/components/schemas/PublicServiceStatus" } + schema: { $ref: "#/components/schemas/PublicPlatformStatus" } "429": description: Shared public-status rate limit exceeded content: diff --git a/internal/handler/status.go b/internal/handler/status.go index 08f94c7..5bf6d0d 100644 --- a/internal/handler/status.go +++ b/internal/handler/status.go @@ -32,9 +32,62 @@ type StatusConfig struct { } type publicStatusChecks interface { - Farm(context.Context) error - VTA(context.Context) error - Platform(context.Context) error + Farm(context.Context) (farmComponentStatus, error) + Platform(context.Context) (platformComponentStatus, error) +} + +type farmComponentStatus struct { + Database string `json:"database"` + Kubernetes string `json:"kubernetes"` + Vault string `json:"vault"` +} + +func unknownFarmComponentStatus() farmComponentStatus { + return farmComponentStatus{ + Database: "unknown", + Kubernetes: "unknown", + Vault: "unknown", + } +} + +func (s *farmComponentStatus) set(component, status string) { + switch component { + case "database": + s.Database = status + case "kubernetes": + s.Kubernetes = status + case "vault": + s.Vault = status + } +} + +type platformComponentStatus struct { + VTA string `json:"vta"` + Mediator string `json:"mediator"` + DIDs string `json:"dids"` + VTC string `json:"vtc"` +} + +func unknownPlatformComponentStatus() platformComponentStatus { + return platformComponentStatus{ + VTA: "unknown", + Mediator: "unknown", + DIDs: "unknown", + VTC: "unknown", + } +} + +func (s *platformComponentStatus) set(component, status string) { + switch component { + case "vta": + s.VTA = status + case "mediator": + s.Mediator = status + case "dids": + s.DIDs = status + case "vtc": + s.VTC = status + } } // StatusHandler exposes customer-safe aggregate health. Detailed failures are @@ -62,24 +115,25 @@ func NewStatusHandler(db *gorm.DB, k8sClient *k8s.Client, cfg StatusConfig) *Sta } func (h *StatusHandler) Farm(c *gin.Context) { - h.respond(c, "farm", h.checks.Farm) -} + ctx, cancel := context.WithTimeout(c.Request.Context(), h.timeout) + defer cancel() -func (h *StatusHandler) VTA(c *gin.Context) { - h.respond(c, "vta", h.checks.VTA) + components, err := h.checks.Farm(ctx) + h.respond(c, "infra", components, err) } func (h *StatusHandler) Platform(c *gin.Context) { - h.respond(c, "platform", h.checks.Platform) -} - -func (h *StatusHandler) respond(c *gin.Context, component string, check func(context.Context) error) { ctx, cancel := context.WithTimeout(c.Request.Context(), h.timeout) defer cancel() + components, err := h.checks.Platform(ctx) + h.respond(c, "platform", components, err) +} + +func (h *StatusHandler) respond(c *gin.Context, component string, components any, err error) { status := "operational" code := http.StatusOK - if err := check(ctx); err != nil { + if err != nil { log.Printf("public status %s unavailable: %v", component, err) status, code = "unavailable", http.StatusServiceUnavailable } @@ -87,6 +141,7 @@ func (h *StatusHandler) respond(c *gin.Context, component string, check func(con c.JSON(code, gin.H{ "status": status, "checked_at": h.now().UTC().Format(time.RFC3339), + "components": components, }) } @@ -97,94 +152,136 @@ type livePublicStatusChecks struct { http *http.Client } -func (h *livePublicStatusChecks) Farm(ctx context.Context) error { - if h.db == nil { - return errors.New("database unavailable") - } - if h.k8s == nil { - return errors.New("kubernetes client unavailable") - } - if h.cfg.VaultAddr == "" { - return errors.New("vault not configured") +func (h *livePublicStatusChecks) Farm(ctx context.Context) (farmComponentStatus, error) { + checks := []statusComponentCheck{ + {component: "database", check: func(componentCtx context.Context) error { + if h.db == nil { + return errors.New("database unavailable") + } + sqlDB, err := h.db.DB() + if err != nil { + return fmt.Errorf("database handle: %w", err) + } + if err := sqlDB.PingContext(componentCtx); err != nil { + return fmt.Errorf("database ping: %w", err) + } + return nil + }}, + {component: "kubernetes", check: func(componentCtx context.Context) error { + if h.k8s == nil { + return errors.New("kubernetes client unavailable") + } + if err := h.k8s.Ping(componentCtx); err != nil { + return fmt.Errorf("kubernetes ping: %w", err) + } + return nil + }}, + {component: "vault", check: func(componentCtx context.Context) error { + if h.cfg.VaultAddr == "" { + return errors.New("vault not configured") + } + status, err := vault.Health(componentCtx, h.cfg.VaultAddr, h.cfg.VaultSkipVerify) + if err != nil { + return fmt.Errorf("vault health: %w", err) + } + if status != "ok" { + return fmt.Errorf("vault state: %s", status) + } + return nil + }}, } - group, groupCtx := errgroup.WithContext(ctx) - group.Go(func() error { - componentCtx, cancel := context.WithTimeout(groupCtx, statusProbeTimeout) - defer cancel() - sqlDB, err := h.db.DB() - if err != nil { - return fmt.Errorf("database handle: %w", err) - } - if err := sqlDB.PingContext(componentCtx); err != nil { - return fmt.Errorf("database ping: %w", err) - } - return nil - }) - group.Go(func() error { - componentCtx, cancel := context.WithTimeout(groupCtx, statusProbeTimeout) - defer cancel() - if err := h.k8s.Ping(componentCtx); err != nil { - return fmt.Errorf("kubernetes ping: %w", err) - } - return nil - }) - group.Go(func() error { - componentCtx, cancel := context.WithTimeout(groupCtx, statusProbeTimeout) - defer cancel() - status, err := vault.Health(componentCtx, h.cfg.VaultAddr, h.cfg.VaultSkipVerify) - if err != nil { - return fmt.Errorf("vault health: %w", err) - } - if status != "ok" { - return fmt.Errorf("vault state: %s", status) - } - return nil - }) - return group.Wait() + components := unknownFarmComponentStatus() + results, err := runStatusChecks(ctx, checks) + for component, status := range results { + components.set(component, status) + } + return components, err } -func (h *livePublicStatusChecks) VTA(ctx context.Context) error { +func (h *livePublicStatusChecks) Platform(ctx context.Context) (platformComponentStatus, error) { + components := unknownPlatformComponentStatus() stack, err := h.platformStack(ctx) if err != nil { - return err + return components, err } - target := strings.TrimRight(stack.PublicURL(), "/") + "/health" - if err := validateStatusProbeURL(target, h.cfg.ClusterDomain); err != nil { - return fmt.Errorf("platform VTA target: %w", err) + targets := []statusProbeTarget{ + {component: "vta", target: strings.TrimRight(stack.PublicURL(), "/") + "/health"}, + {component: "mediator", target: "https://" + stack.MediatorFQDN() + "/mediator/v1/readyz"}, + {component: "dids", target: strings.TrimRight(stack.DidsURL(), "/") + "/health"}, + {component: "vtc", target: "https://" + stack.VtcFQDN() + "/health"}, } - return h.probe(ctx, target) -} -func (h *livePublicStatusChecks) Platform(ctx context.Context) error { - stack, err := h.platformStack(ctx) - if err != nil { - return err + validChecks := make([]statusComponentCheck, 0, len(targets)) + failures := make([]error, 0, len(targets)) + for _, probe := range targets { + if err := validateStatusProbeURL(probe.target, h.cfg.ClusterDomain); err != nil { + components.set(probe.component, "unavailable") + failures = append(failures, fmt.Errorf("%s target: %w", probe.component, err)) + continue + } + target := probe.target + validChecks = append(validChecks, statusComponentCheck{ + component: probe.component, + check: func(componentCtx context.Context) error { + return h.probe(componentCtx, target) + }, + }) } - targets := map[string]string{ - "mediator": "https://" + stack.MediatorFQDN() + "/mediator/v1/readyz", - "dids": strings.TrimRight(stack.DidsURL(), "/") + "/health", - "vtc": "https://" + stack.VtcFQDN() + "/health", + results, probeErr := runStatusChecks(ctx, validChecks) + for component, status := range results { + components.set(component, status) } - for component, target := range targets { - if err := validateStatusProbeURL(target, h.cfg.ClusterDomain); err != nil { - return fmt.Errorf("%s target: %w", component, err) - } + if probeErr != nil { + failures = append(failures, probeErr) } + return components, errors.Join(failures...) +} + +type statusProbeTarget struct { + component string + target string +} + +type statusComponentCheck struct { + component string + check func(context.Context) error +} + +type statusComponentResult struct { + component string + err error +} + +func runStatusChecks(ctx context.Context, checks []statusComponentCheck) (map[string]string, error) { + results := make(chan statusComponentResult, len(checks)) - group, groupCtx := errgroup.WithContext(ctx) - for component, target := range targets { - component, target := component, target + var group errgroup.Group + for _, check := range checks { + check := check group.Go(func() error { - if err := h.probe(groupCtx, target); err != nil { - return fmt.Errorf("%s: %w", component, err) - } + componentCtx, cancel := context.WithTimeout(ctx, statusProbeTimeout) + defer cancel() + results <- statusComponentResult{component: check.component, err: check.check(componentCtx)} return nil }) } - return group.Wait() + _ = group.Wait() + close(results) + + statuses := make(map[string]string, len(checks)) + failures := make([]error, 0, len(checks)) + for result := range results { + status := "operational" + if result.err != nil { + status = "unavailable" + failures = append(failures, fmt.Errorf("%s: %w", result.component, result.err)) + } + statuses[result.component] = status + } + return statuses, errors.Join(failures...) } func (h *livePublicStatusChecks) platformStack(ctx context.Context) (*model.SetupSession, error) { diff --git a/internal/handler/status_test.go b/internal/handler/status_test.go index c05f8fb..bff8732 100644 --- a/internal/handler/status_test.go +++ b/internal/handler/status_test.go @@ -7,6 +7,7 @@ import ( "net/http" "net/http/httptest" "strings" + "sync/atomic" "testing" "time" @@ -14,29 +15,61 @@ import ( ) type fakePublicStatusChecks struct { - farmErr error - vtaErr error - platformErr error + farmComponents farmComponentStatus + farmErr error + platformComponents platformComponentStatus + platformErr error } -func (f *fakePublicStatusChecks) Farm(context.Context) error { return f.farmErr } -func (f *fakePublicStatusChecks) VTA(context.Context) error { return f.vtaErr } -func (f *fakePublicStatusChecks) Platform(context.Context) error { return f.platformErr } +func (f *fakePublicStatusChecks) Farm(context.Context) (farmComponentStatus, error) { + return f.farmComponents, f.farmErr +} +func (f *fakePublicStatusChecks) Platform(context.Context) (platformComponentStatus, error) { + return f.platformComponents, f.platformErr +} func TestStatusHandlerResponses(t *testing.T) { gin.SetMode(gin.TestMode) fixed := time.Date(2026, 10, 24, 23, 0, 0, 0, time.FixedZone("test", -7*60*60)) tests := []struct { - name string - path string - configure func(*fakePublicStatusChecks) - wantCode int - wantStatus string + name string + path string + configure func(*fakePublicStatusChecks) + wantCode int + wantStatus string + wantComponents map[string]string }{ - {name: "farm operational", path: "/farm", configure: func(*fakePublicStatusChecks) {}, wantCode: http.StatusOK, wantStatus: "operational"}, - {name: "VTA unavailable", path: "/vta", configure: func(f *fakePublicStatusChecks) { f.vtaErr = errors.New("private failure detail") }, wantCode: http.StatusServiceUnavailable, wantStatus: "unavailable"}, - {name: "platform unavailable", path: "/platform", configure: func(f *fakePublicStatusChecks) { f.platformErr = errors.New("private failure detail") }, wantCode: http.StatusServiceUnavailable, wantStatus: "unavailable"}, + { + name: "farm operational", path: "/farm", wantCode: http.StatusOK, wantStatus: "operational", + configure: func(f *fakePublicStatusChecks) { + f.farmComponents = farmComponentStatus{Database: "operational", Kubernetes: "operational", Vault: "operational"} + }, + wantComponents: map[string]string{"database": "operational", "kubernetes": "operational", "vault": "operational"}, + }, + { + name: "farm exposes sanitized component failures", path: "/farm", wantCode: http.StatusServiceUnavailable, wantStatus: "unavailable", + configure: func(f *fakePublicStatusChecks) { + f.farmComponents = farmComponentStatus{Database: "unavailable", Kubernetes: "operational", Vault: "unavailable"} + f.farmErr = errors.New("private failure detail") + }, + wantComponents: map[string]string{"database": "unavailable", "kubernetes": "operational", "vault": "unavailable"}, + }, + { + name: "platform operational", path: "/platform", wantCode: http.StatusOK, wantStatus: "operational", + configure: func(f *fakePublicStatusChecks) { + f.platformComponents = platformComponentStatus{VTA: "operational", Mediator: "operational", DIDs: "operational", VTC: "operational"} + }, + wantComponents: map[string]string{"vta": "operational", "mediator": "operational", "dids": "operational", "vtc": "operational"}, + }, + { + name: "platform exposes sanitized component failures", path: "/platform", wantCode: http.StatusServiceUnavailable, wantStatus: "unavailable", + configure: func(f *fakePublicStatusChecks) { + f.platformComponents = platformComponentStatus{VTA: "unavailable", Mediator: "unavailable", DIDs: "operational", VTC: "operational"} + f.platformErr = errors.New("private failure detail") + }, + wantComponents: map[string]string{"vta": "unavailable", "mediator": "unavailable", "dids": "operational", "vtc": "operational"}, + }, } for _, tt := range tests { @@ -46,7 +79,6 @@ func TestStatusHandlerResponses(t *testing.T) { h := &StatusHandler{checks: checks, timeout: time.Second, now: func() time.Time { return fixed }} router := gin.New() router.GET("/farm", h.Farm) - router.GET("/vta", h.VTA) router.GET("/platform", h.Platform) resp := httptest.NewRecorder() @@ -61,7 +93,20 @@ func TestStatusHandlerResponses(t *testing.T) { if body["status"] != tt.wantStatus || body["checked_at"] != "2026-10-25T06:00:00Z" { t.Fatalf("body = %#v", body) } - if len(body) != 2 { + wantFields := 2 + if tt.wantComponents != nil { + wantFields = 3 + components, ok := body["components"].(map[string]any) + if !ok { + t.Fatalf("components = %#v", body["components"]) + } + for component, want := range tt.wantComponents { + if components[component] != want { + t.Errorf("components[%q] = %#v, want %q", component, components[component], want) + } + } + } + if len(body) != wantFields { t.Fatalf("public response leaked extra fields: %#v", body) } if resp.Header().Get("Cache-Control") != "no-store" { @@ -74,6 +119,44 @@ func TestStatusHandlerResponses(t *testing.T) { } } +func TestRunStatusChecksCollectsConcurrentFailures(t *testing.T) { + var arrived atomic.Int32 + ready := make(chan struct{}) + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if arrived.Add(1) == 4 { + close(ready) + } + select { + case <-ready: + case <-r.Context().Done(): + return + } + if r.URL.Path == "/vta" || r.URL.Path == "/mediator" { + http.Error(w, "down", http.StatusServiceUnavailable) + return + } + w.WriteHeader(http.StatusNoContent) + })) + defer server.Close() + + checks := &livePublicStatusChecks{http: server.Client()} + components, err := runStatusChecks(t.Context(), []statusComponentCheck{ + {component: "vta", check: func(ctx context.Context) error { return checks.probe(ctx, server.URL+"/vta") }}, + {component: "mediator", check: func(ctx context.Context) error { return checks.probe(ctx, server.URL+"/mediator") }}, + {component: "dids", check: func(ctx context.Context) error { return checks.probe(ctx, server.URL+"/dids") }}, + {component: "vtc", check: func(ctx context.Context) error { return checks.probe(ctx, server.URL+"/vtc") }}, + }) + if err == nil || !strings.Contains(err.Error(), "vta:") || !strings.Contains(err.Error(), "mediator:") { + t.Fatalf("error = %v, want both VTA and mediator failures", err) + } + want := map[string]string{"vta": "unavailable", "mediator": "unavailable", "dids": "operational", "vtc": "operational"} + for component, wantStatus := range want { + if components[component] != wantStatus { + t.Errorf("components[%q] = %q, want %q", component, components[component], wantStatus) + } + } +} + func TestStatusProbe(t *testing.T) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { switch r.URL.Path { diff --git a/internal/router/router.go b/internal/router/router.go index 30d4f07..62e4764 100644 --- a/internal/router/router.go +++ b/internal/router/router.go @@ -94,13 +94,12 @@ func Setup( VaultAddr: cfg.Vault.Addr, VaultSkipVerify: cfg.Vault.SkipVerify, }) - // Three one-minute UptimeRobot checks consume about three requests per + // Two one-minute UptimeRobot checks consume about two requests per // minute. Thirty leaves room for retries and manual checks without letting // one source continuously fan out into dependency probes. status := v1.Group("/status", middleware.NoStore(), middleware.RateLimit(30, time.Minute)) { status.GET("/farm", statusHandler.Farm) - status.GET("/vta", statusHandler.VTA) status.GET("/platform", statusHandler.Platform) } From 51adc12223709ae207f09c7d5d20083b3f94fc31 Mon Sep 17 00:00:00 2001 From: vthwang Date: Sun, 25 Oct 2026 15:39:51 -0700 Subject: [PATCH 2/3] chore: release 0.15.2 Publish the consolidated two-monitor status API and its sanitized component health responses. - document the standalone VTA status endpoint removal - bump chart and application versions together Signed-off-by: vthwang --- CHANGELOG.md | 20 ++++++++++++++++++++ helm/vtafarm-api/Chart.yaml | 4 ++-- 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 46126ad..e94cd03 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,25 @@ # Changelog +## [v0.15.2] - 2026-10-25 + +### Breaking + +- Removed the standalone `/api/v1/status/vta` endpoint; VTA availability is + now included in `/api/v1/status/platform`. + +### Added + +- Public infrastructure status reports sanitized Database, Kubernetes, and + Vault component states in one response. +- Public platform status reports sanitized VTA, Mediator, DID Hosting, and VTC + component states while collecting concurrent failures without fail-fast + cancellation. + +### Changed + +- Consolidated the customer-facing status surface into VTA Farm Infra and VTA + Farm Platform monitors. + ## [v0.15.1] - 2026-10-25 ### Added diff --git a/helm/vtafarm-api/Chart.yaml b/helm/vtafarm-api/Chart.yaml index 5086694..ca406eb 100644 --- a/helm/vtafarm-api/Chart.yaml +++ b/helm/vtafarm-api/Chart.yaml @@ -3,5 +3,5 @@ name: vtafarm-api description: VTA Farm API — Go REST backend for Kubernetes pod management type: application # Kept equal; one release bumps both. -version: 0.15.1 -appVersion: "0.15.1" +version: 0.15.2 +appVersion: "0.15.2" From 386afb1dc4bfc009edb7c283ad9664a26405dadc Mon Sep 17 00:00:00 2001 From: vthwang Date: Sun, 25 Oct 2026 15:41:28 -0700 Subject: [PATCH 3/3] docs: refine 0.15.2 release notes Keep the release notes focused on API behavior and the breaking endpoint change. Signed-off-by: vthwang --- CHANGELOG.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e94cd03..c2f4f57 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,11 +15,6 @@ component states while collecting concurrent failures without fail-fast cancellation. -### Changed - -- Consolidated the customer-facing status surface into VTA Farm Infra and VTA - Farm Platform monitors. - ## [v0.15.1] - 2026-10-25 ### Added