handle Fleet Node outages as miner unavailability

[ankitgoswami]

Reviewable diff: +396/-83 across 22 files (excludes generated, test, and story files).

Summary

Miners assigned to a previously connected but unreachable Fleet Node now show Unavailable with No connection instead of presenting stale healthy data. The same outage classification is used by fleet lists, filters, counts, rack slots, CSV exports, and persisted uptime history. Standalone and HA deployments receive a critical Fleet Node outage alert using five non-secret database columns.

How it works

1. Fleet Nodes send heartbeats at most 30 seconds apart. A confirmed node is considered unavailable after its last heartbeat is more than two minutes old; a node that has never connected remains Never connected and does not trigger outage behavior.
2. Miner reads derive an effective OFFLINE status for devices assigned to that node without changing the stored miner status. List, count, filter, selector, rack-slot, CSV, and historical-snapshot paths apply the same rule.
3. The API returns the existing OFFLINE status plus DEVICE_OFFLINE_REASON_FLEET_NODE_UNAVAILABLE. The client maps that reason to Unavailable and No connection while retaining existing offline action and measurement behavior.
4. Both standalone and HA Grafana rule files query fleet_node through a five-column SELECT grant. After the node has been unavailable for another three minutes, Grafana fires a generic critical alert without exposing node names or key material. In HA, both local Grafanas evaluate the rule, while only the active Fleet API accepts webhook delivery.
5. When heartbeats resume, the derived reason disappears, miners return to their last stored status, and Grafana resolves the alert.

flowchart LR
    FN["Fleet Node heartbeat"] --> STATUS["Effective miner status"]
    STATUS --> API["OFFLINE plus reason enum"]
    API --> UI["Unavailable and No connection"]
    API --> VIEWS["Lists, racks, CSV, counts, and selectors"]
    STATUS --> HISTORY["Persisted uptime snapshot"]
    FN --> RULES["Standalone and HA Grafana rules"]
    RULES --> ALERT["Generic outage alert and recovery"]

Loading

stateDiagram-v2
    [*] --> NeverConnected
    NeverConnected --> Available: first heartbeat
    Available --> Unavailable: last heartbeat older than two minutes
    Unavailable --> Alerting: unavailable for three more minutes
    Unavailable --> Available: heartbeat resumes
    Alerting --> Available: heartbeat resumes and alert resolves

Loading

Areas of the code involved

Area / package / file	What changed	Why it matters for review
proto/fleetmanagement/	Adds DeviceOfflineReason and MinerStateSnapshot.offline_reason	Review the additive API contract and reason semantics
server/internal/domain/stores/sqlstores/ and server/sqlc/queries/	Derives Fleet Node-aware effective status across reads, filters, counts, selectors, rack slots, and persisted snapshots	Review consistency, first-heartbeat semantics, and recovery behavior
server/internal/domain/fleetmanagement/	Maps the derived reason into API snapshots and CSV copy	Review the boundary between stored and derived state
client/src/protoFleet/features/	Renders Unavailable, No connection, and the matching two-minute Nodes-page threshold	Review operator-visible behavior
server/migrations/000145_* and deployment-files/run-fleet.sh	Grants Grafana only org_id, id, last_seen_at, enrollment_status, and deleted_at	Review standalone and HA least-privilege provisioning
server/monitoring/grafana/ and server/internal/domain/alerts/	Provisions the outage rule for standalone and HA and adds generic firing/recovery copy	Review the five-minute alert lifecycle and HA delivery behavior
server/internal/handlers/alerts/	Treats the generic Fleet Node template as safe for alert-only readers	Review the permission boundary
server/cmd/fleetnode/	Enforces a maximum 30-second heartbeat interval	Review compatibility for custom launch flags
**/generated/**, *.pb.go, *.pb.ts	Regenerated protobuf and sqlc output	Generated — skip

Key technical decisions & trade-offs

• Keep DeviceStatus.OFFLINE and add a reason enum instead of creating a Fleet Node-specific status, preserving existing offline behavior while allowing precise copy.
• Derive outages at read and snapshot time instead of persisting a replacement device status, so recovery restores the miner's last reported state without a repair write.
• Reuse the existing effective-status fragment in dynamic Go SQL, while keeping static sqlc and Grafana predicates explicit instead of introducing a database function or configuration abstraction.
• Require a real last_seen_at; confirmation alone does not turn a never-connected node into an outage.
• Grant Grafana five lifecycle columns instead of table access or a new database view/function; names and key material remain denied.
• Provision the rule directly in both standalone and HA rule files rather than mounting the full standalone rule set into HA.
• Fail fast when a custom heartbeat interval exceeds 30 seconds, keeping the two-minute outage threshold bounded instead of adding clamping or another runtime setting.
• Apply the migration grant only when the HA Grafana role already exists, and repeat the grant in run-fleet.sh for roles created after migrations.

Testing & validation

• just gen is stable after regenerating protobuf and sqlc output.
• just lint passes: buf, ESLint, and golangci-lint report no issues.
• Focused Go tests pass for CSV copy, Fleet Node rule provisioning, exact grant/revoke columns, dynamic/static staleness SQL, and heartbeat validation.
• bash -n deployment-files/run-fleet.sh and git diff --check pass.
• The existing database integration test now covers live outage/recovery, persisted offline history, rack offline counts, and rack-slot status, but was not run locally; CI owns the database-backed validation.
• No live Grafana firing/recovery cycle, role-level privilege integration test, or production-scale query-plan benchmark was run.

Post-Deploy Monitoring & Validation

• Window / owner: The deployer or Fleet on-call watches the first 30 minutes after rollout and the first controlled Fleet Node outage/recovery in standalone and HA.
• Logs: Search Fleet API and Grafana logs for protofleet-fleet-node-unavailable, permission denied for table fleet_node, query errors, and alert evaluation errors.
• Expected behavior: A node that has heartbeated should move its miners to Unavailable after two minutes and fire one alert after five minutes total. A confirmed node that has never heartbeated should remain Never connected without an outage alert.
• Healthy recovery: After the node resumes heartbeats, live views and subsequent history snapshots return to the stored miner status and Grafana resolves the alert.
• Failure / mitigation trigger: Roll back or disable the rule if healthy nodes are marked unavailable, counts disagree with rack/list/history views, HA duplicates delivered notifications, Grafana reports permission errors, the alert flaps, or recovery does not clear the reason.

[github-actions]

🔐 Codex Security Review

Note: This is an automated security-focused code review generated by Codex.
It should be used as a supplementary check alongside human review.
False positives are possible - use your judgment.

Scope summary

• Reviewed pull request diff only (d5504553ab2088052762803907ef6d286e72d40d...8e75ce9405143a1233e5dd6e135e8644c182d7a4, exact PR three-dot diff)
• Model: gpt-5.6-sol

💡 Click "edited" above to see previous reviews for this PR.

---

Review Summary

Overall Risk: MEDIUM

Findings

[MEDIUM] Persisted uptime ignores Fleet Node outages

• Category: Reliability
• Location: server/sqlc/queries/device.sql:341
• Description: Live counts now classify miners behind a stale Fleet Node using effective_status, but InsertMinerStateSnapshot still classifies them using the last raw device_status. An active miner therefore remains recorded as hashing throughout a Fleet Node outage while the live list reports it offline.
• Impact: Historical uptime, availability reporting, and outage postmortems become inaccurate; the outage disappears from history after the live bucket ages out.
• Recommendation: Apply the same Fleet Node staleness calculation to InsertMinerStateSnapshot and add an integration test covering outage and recovery snapshots.

[MEDIUM] Rack slots remain healthy while rack counts report offline

• Category: Reliability
• Location: server/internal/domain/stores/sqlstores/device.go:1476
• Description: Collection counts now use effective_status, but GetRackSlotStatuses still uses raw device_status. During a Fleet Node outage, a rack can report an increased offline count while the affected slots remain marked healthy.
• Impact: Operators receive contradictory rack-health data and cannot reliably identify affected miners from the rack visualization.
• Recommendation: Reuse the effective-status expression in the rack-slot query and test a stale Fleet Node whose assigned miner last reported ACTIVE.

Notes

The protobuf change is additive and the changed dynamic SQL remains parameterized. No authentication, injection, credential exposure, or pool-hijacking issue was identified in the reviewed hunks. git diff --check passed; repository test commands were not run because just is unavailable in the review environment.

---

_{Generated by Codex Security Review |
Triggered by: @ankitgoswami |
Review workflow run}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8e75ce9405

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[Copilot]

🔵 Needs a closer look

It is a cross-cutting change spanning protobuf contracts, SQL query semantics, alerting/provisioning, and UI behavior, so it merits final human review for operational correctness and performance implications.

Pull request overview

Implements Fleet Node outage awareness across the stack so miners assigned to an unreachable Fleet Node are treated as effectively OFFLINE while preserving the stored device status, enabling consistent filtering/counting behavior and clearer UI messaging (“Unavailable” / “No connection”), plus a least-privilege Grafana alert for node heartbeat staleness.

Changes:

• Derives an effective miner OFFLINE status when the assigned Fleet Node is stale, and threads that effective status through list/filter/count/selector query paths.
• Extends the Fleet Management API with DeviceOfflineReason and maps Fleet Node staleness to DEVICE_OFFLINE_REASON_FLEET_NODE_UNAVAILABLE.
• Adds a Grafana alert rule for Fleet Node unavailability with column-level DB grants that avoid exposing node names/keys.

File summaries

File	Description
server/sqlc/queries/device.sql	Updates sqlc query logic to use an effective status that treats stale Fleet Nodes as OFFLINE.
server/monitoring/grafana/provisioning/alerting/proto-fleet-rules.yaml	Adds the “Fleet Node Unavailable” alert rule using fleet_node heartbeat staleness.
server/migrations/000145_grant_fleet_node_alert_access.up.sql	Grants Grafana HA read-only role column-level access needed for the outage rule.
server/migrations/000145_grant_fleet_node_alert_access.down.sql	Revokes the column-level access granted in the up migration.
server/internal/infrastructure/timescaledb/provisioned_rules_test.go	Ensures the packaged default rule set includes “Fleet Node Unavailable”.
server/internal/infrastructure/timescaledb/fleet_node_rule_test.go	Validates alert rule semantics (staleness window, UID length) and grant column allowlist.
server/internal/infrastructure/db/migration_bridges_test.go	Bumps expected migration version to 145.
server/internal/handlers/alerts/handler.go	Treats Fleet Node outage template as device-less/safe for alert-only readers and maps it to proto “unspecified”.
server/internal/handlers/alerts/handler_history_test.go	Adds coverage for visibility of Fleet Node outage summary without miner:read.
server/internal/domain/stores/sqlstores/device.go	Threads FleetNodeUnavailable through scan/mapping and uses effective status in state count paths.
server/internal/domain/stores/sqlstores/device_query_fragments.go	Introduces fleetNodeUnavailableExpr and effectiveDeviceStatusExpr and adds Fleet Node joins into common miner query fragments.
server/internal/domain/stores/sqlstores/device_query_builder_test.go	Adds tests asserting Fleet Node outages are derived as OFFLINE and shared across list/count queries.
server/internal/domain/stores/sqlstores/device_integration_test.go	Adds DB integration test verifying outage derivation and recovery behavior.
server/internal/domain/stores/sqlstores/device_filters.go	Switches filter predicates to use effective_status.status (including numeric-range offline exclusion).
server/internal/domain/stores/sqlstores/device_filters_test.go	Updates/extends tests to assert effective-status SQL output.
server/internal/domain/fleetmanagement/service.go	Maps FleetNodeUnavailable to OFFLINE + DEVICE_OFFLINE_REASON_FLEET_NODE_UNAVAILABLE.
server/internal/domain/fleetmanagement/service_test.go	Adds assertions that the new offline reason is set when FleetNodeUnavailable is true.
server/internal/domain/alerts/service.go	Parses the new fleet-node-unavailable template label into a typed RuleTemplate.
server/internal/domain/alerts/render.go	Adds generic resolved copy for Fleet Node outage alerts.
server/internal/domain/alerts/render_test.go	Adds Slack rendering coverage for generic Fleet Node outage copy.
server/internal/domain/alerts/models.go	Defines RuleTemplateFleetNodeUnavailable.
server/generated/sqlc/device.sql.go	Generated sqlc updates reflecting effective-status logic and new FleetNodeUnavailable column.
server/generated/grpc/fleetmanagement/v1/fleetmanagement.pb.go	Generated protobuf updates adding DeviceOfflineReason and offline_reason field.
server/cmd/fleetnode/run.go	Validates heartbeat interval and caps it at the 30s default maximum.
server/cmd/fleetnode/run_test.go	Adds tests for heartbeat interval validation behavior.
proto/fleetmanagement/v1/fleetmanagement.proto	Adds DeviceOfflineReason and MinerStateSnapshot.offline_reason.
deployment-files/run-fleet.sh	Extends Grafana provisioning to wait for fleet_node and grant only required columns; adds a smoke-check query.
client/src/protoFleet/features/settings/components/Nodes/NodeStatusBadge.tsx	Updates “stale” threshold to 2 minutes to match server-side outage semantics.
client/src/protoFleet/features/settings/components/Nodes/NodeStatusBadge.test.tsx	Updates badge boundary tests to match the new 2-minute threshold.
client/src/protoFleet/features/fleetManagement/components/MinerList/stories/statusMocks.ts	Updates Storybook mocks to include offlineReason.
client/src/protoFleet/features/fleetManagement/components/MinerList/stories/mocks.ts	Updates Storybook mocks to include offlineReason.
client/src/protoFleet/features/fleetManagement/components/MinerList/MinerStatus.tsx	Renders “Unavailable” when offlineReason indicates Fleet Node unavailability.
client/src/protoFleet/features/fleetManagement/components/MinerList/MinerStatus.test.tsx	Adds coverage for “Unavailable” rendering based on offlineReason.
client/src/protoFleet/features/fleetManagement/components/MinerList/MinerIssues.tsx	Renders “No connection” and suppresses other issue UI when offlineReason indicates Fleet Node unavailability.
client/src/protoFleet/features/fleetManagement/components/MinerList/MinerIssues.test.tsx	Adds coverage for “No connection” and icon/skeleton suppression for Fleet Node outages.
client/src/protoFleet/api/generated/fleetmanagement/v1/fleetmanagement_pb.ts	Generated TS protobuf updates adding offlineReason enum/field.

Review details

• Files reviewed: 33/36 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ankitgoswami]

🤖

Persisted uptime ignores Fleet Node outages

Rack slots remain healthy while rack counts report offline

Addressed in 14f7f68d2: persisted snapshots now use effective Fleet Node status, and rack slots reuse the same effective-status expression. The existing outage integration test now covers persisted offline history, rack offline counts, and the corresponding offline slot.