Skip to content

OTA: host canary health-gate + commanded rollback on bad firmware #33

Description

@TalkingJupiter

Design note surfaced while planning OTA (#27). After a device updates and confirms itself (MQTT connected), it can still be a healthy-but-misbehaving image: it boots and connects but a logic bug breaks its actual behavior, for example telemetry stops or goes wrong. The automatic bootloader rollback does not catch this. The host, which sees telemetry and heartbeat, is the component that can detect it and remediate.

Scope (host side, fleet phases)

  • After an update, watch the updated (canary) device: telemetry flowing and well-formed, heartbeat healthy, and the reported firmware version equals the target.
  • If it passes: proceed. In a rack this means update the standby first, verify, then the primary, never both at once.
  • If it fails: issue a commanded rollback to stabilize the device (see the firmware-side rollback issue), and notify the admin, rather than only alerting.
  • Optional: a manual approval gate before rolling to the rest of the fleet (safer early on, can be automated once the pipeline is trusted).

Relates to

Part of #27.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions