feat(sysupdate): reboot coordination for non-Kubernetes and single-node hosts

docs/skills/architecture-roadmap.md

@@ -27,7 +27,7 @@ Priorities are derived from [gap-analysis-distros.md](gap-analysis-distros.md).
 | 3 | Boot-time selection / automatic rollback of a failed update | No previous OS version is kept once a root update overwrites the slot. |
 | 4 | Broader `systemd-creds` integration for SSH keys and network configuration | Only `root` password credential path is shipped in `os-creds-prov.bst`. |
 | 5 | TPM2-bound credential delivery at first boot | Documented but not wired into the installed OS image. |
-| 6 | Native reboot coordination for non-Kubernetes and single-node hosts | Kured only covers Kubernetes nodes; no FleetLock/locksmith equivalent. |
+| 6 | Distributed reboot lock coordination for multi-node non-Kubernetes clusters | Single-node / maintenance window / lock-file reboot coordination is present via `systemd-sysupdate-reboot`; cluster-wide FleetLock/locksmith HTTP protocol is not implemented. |
 | 7 | Staged rollout behavior for larger fleets | Future after items 1-3 are implemented. |
 
 ## Status notes

docs/skills/gap-analysis-distros.md

@@ -99,7 +99,7 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from
 | **Updates** | `systemd-sysupdate` reads root/UKI transfers from `files/os/sysupdate.d/` and the optional k0s transfer from the `k0s` component directory. Assets are published to GitHub Releases, and the combined `SHA256SUMS` manifest is signed in CI with a GPG key. `Verify=yes` is the default. Sources: [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md), [50-root.transfer](../../files/os/sysupdate.d/50-root.transfer), [60-uki.transfer](../../files/os/sysupdate.d/60-uki.transfer), [70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer), also `systemd-sysupdate(8)`. |
 | **Provisioning** | The installer is an offline `systemd-sysinstall` image that embeds the DDI as a data partition. First-boot configuration is intended to be delivered via `systemd-creds` through the ESP or hypervisor metadata. Today only `passwd.hashed-password.root` is consumed via `systemd-sysusers.d`; the documented `tmpfiles.extra` path for SSH keys and similar files is not implemented. Sources: [bluefin-server-installer.bst](../../elements/oci/bluefin-server-installer.bst), [10-root-creds.conf](../../files/os/sysusers.d/10-root-creds.conf), [os-creds-prov.bst](../../elements/bluefin-server/os-creds-prov.bst), [systemd-creds(1)](https://www.freedesktop.org/software/systemd/man/latest/systemd-creds.html). |
 | **Customization** | Adds software through `systemd-sysext` (overlay `/usr`) and `systemd-confext` (overlay `/etc`) images. The base OS `os-release` advertises `ID=flatcar` and a matching `VERSION_ID` so pre-built Flatcar Bakery extensions load. k0s is shipped as a separately built, optionally enabled sysext. Sources: [systemd-sysext-extensions.md](systemd-sysext-extensions.md), [k0s-sysext.md](k0s-sysext.md), [os-release-flatcar.bst](../../elements/bluefin-server/os-release-flatcar.bst), [systemd-sysext(8)](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html). |
-| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStartPost` that touches `/run/reboot-required`. Rolling reboots across Kubernetes nodes rely on Kured reading that file. Sources: [os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst), [kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf), [Kured project](https://github.com/weaveworks/kured). |
+| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStartPost` that touches `/run/reboot-required` for Kured. For non-Kubernetes and single-node hosts, `systemd-sysupdate-reboot.timer` and `systemd-sysupdate-reboot.service` coordinate automated reboots using systemd primitives, supporting maintenance window timers and lock inhibition (`/run/reboot-lock`, `/etc/reboot-lock`, `/run/kured-active`). Sources: [os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst), [kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf), [systemd-sysupdate-reboot.service](../../files/os/systemd/system/systemd-sysupdate-reboot.service), [systemd-sysupdate-reboot.timer](../../files/os/systemd/system/systemd-sysupdate-reboot.timer), [Kured project](https://github.com/weaveworks/kured). |
 
 ## 4. Factual Gaps
 
@@ -120,15 +120,15 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from
 ### Update delivery
 
 - **Gap:** The root transfer uses `Type=partition Path=auto`, which requires `systemd-sysupdate` to discover a matching GPT partition label (`bluefin-server-root-a`/`root-b`). This is correct, but without a `root-b` partition the transfer effectively overwrites the running root in place.
-- **Gap:** `systemd-sysupdate-reboot.service`/`systemd-sysupdate-reboot.timer` are not enabled or configured; the only reboot signal today is the Kured hook.
 
 ### Customization
 
 - **No major gap found relative to the design intent.** `systemd-sysext` and Flatcar Bakery compatibility match the intended extension model. The main operational concern is that, because `/usr` is not mounted read-only, a sysext merge vs. runtime writes to `/usr` have different guarantees than on Flatcar or Fedora CoreOS.
 
 ### Reboot coordination
 
-- **Gap:** Kured coordinates Kubernetes node reboots but requires Kubernetes to be running. There is no equivalent for single-node or non-Kubernetes Bluefin hosts, and there is no built-in cluster lock manager similar to Zincati's FleetLock or Flatcar's `locksmithd`/`etcd-lock`.
+- **Status:** Bluefin Server provides `systemd-sysupdate-reboot.timer` and `systemd-sysupdate-reboot.service` with maintenance window scheduling and lock-based inhibition (`/run/reboot-lock`, `/etc/reboot-lock`, `/run/kured-active`) for single-node and non-Kubernetes hosts, alongside the Kured hook for Kubernetes clusters.
+- **Gap:** While single-node and lock-file inhibition is supported via systemd primitives, there is no built-in cluster-wide HTTP lock manager client similar to Zincati's FleetLock or Flatcar's `locksmithd`/`etcd-lock`.
 
 ## 5. Summary of Biggest Gaps
 
@@ -195,3 +195,6 @@ These gaps drive the priorities in [architecture-roadmap.md](architecture-roadma
 - [files/os/sysupdate.k0s.d/70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer)
 - [files/os/sysusers.d/10-root-creds.conf](../../files/os/sysusers.d/10-root-creds.conf)
 - [files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf)
+- [files/os/systemd/system/systemd-sysupdate-reboot.service](../../files/os/systemd/system/systemd-sysupdate-reboot.service)
+- [files/os/systemd/system/systemd-sysupdate-reboot.timer](../../files/os/systemd/system/systemd-sysupdate-reboot.timer)
+- [files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset](../../files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset)

files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset

@@ -0,0 +1,2 @@
+# Enable automatic reboot timer for system updates on non-Kubernetes and single-node hosts
+enable systemd-sysupdate-reboot.timer

files/os/systemd/system/systemd-sysupdate-reboot.service

@@ -0,0 +1,14 @@
+[Unit]
+Description=Reboot Automatically After System Update
+Documentation=man:systemd-sysupdate-reboot.service(8)
+ConditionVirtualization=!container
+ConditionPathExists=!/run/reboot-lock
+ConditionPathExists=!/etc/reboot-lock
+ConditionPathExists=!/run/kured-active
+
+[Service]
+Type=oneshot
+ExecStart=/usr/bin/systemd-sysupdate reboot
+
+[Install]
+Also=systemd-sysupdate-reboot.timer

files/os/systemd/system/systemd-sysupdate-reboot.timer

@@ -0,0 +1,12 @@
+[Unit]
+Description=Reboot Automatically After System Update
+Documentation=man:systemd-sysupdate-reboot.service(8)
+ConditionVirtualization=!container
+
+[Timer]
+OnCalendar=*-*-* 04:10:00
+RandomizedDelaySec=30min
+Persistent=true
+
+[Install]
+WantedBy=timers.target

tests/unit/test_reboot_coordination.py

@@ -0,0 +1,113 @@
+"""Unit tests for systemd-sysupdate reboot coordination.
+
+Asserts contracts for systemd-sysupdate-reboot service, timer, presets,
+and drop-in coordination strategies (immediate, maintenance window, lock-based)
+while ensuring Kured reboot paths are preserved without collisions.
+"""
+
+import configparser
+from pathlib import Path
+import pytest
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+SYSTEMD_SYSTEM_DIR = REPO_ROOT / "files" / "os" / "systemd" / "system"
+SYSTEMD_PRESET_DIR = REPO_ROOT / "files" / "os" / "systemd" / "system-preset"
+SYSUPDATE_DROPIN_DIR = (
+    REPO_ROOT / "files" / "os" / "systemd" / "systemd-sysupdate.service.d"
+)
+ELEMENTS_DIR = REPO_ROOT / "elements" / "bluefin-server"
+OS_STACK = ELEMENTS_DIR / "os-stack.bst"
+
+REBOOT_SERVICE = SYSTEMD_SYSTEM_DIR / "systemd-sysupdate-reboot.service"
+REBOOT_TIMER = SYSTEMD_SYSTEM_DIR / "systemd-sysupdate-reboot.timer"
+REBOOT_PRESET = SYSTEMD_PRESET_DIR / "zz-enable-sysupdate-reboot.preset"
+KURED_HOOK = SYSUPDATE_DROPIN_DIR / "kured-hook.conf"
+
+
+def load_ini(path: Path) -> configparser.ConfigParser:
+    parser = configparser.ConfigParser(strict=False)
+    parser.optionxform = str
+    parser.read_string(path.read_text())
+    return parser
+
+
+def test_reboot_service_and_timer_exist():
+    assert REBOOT_SERVICE.is_file(), "systemd-sysupdate-reboot.service is missing"
+    assert REBOOT_TIMER.is_file(), "systemd-sysupdate-reboot.timer is missing"
+    assert REBOOT_PRESET.is_file(), "zz-enable-sysupdate-reboot.preset is missing"
+
+
+def test_reboot_service_contract():
+    parser = load_ini(REBOOT_SERVICE)
+    assert parser.has_section("Unit")
+    assert parser.has_section("Service")
+    assert parser.has_section("Install")
+
+    unit = parser["Unit"]
+    assert unit.get("ConditionVirtualization") == "!container"
+
+    # Verify lock-based coordination and kured coordination guards
+    content = REBOOT_SERVICE.read_text()
+    assert "ConditionPathExists=!/run/reboot-lock" in content
+    assert "ConditionPathExists=!/etc/reboot-lock" in content
+    assert "ConditionPathExists=!/run/kured-active" in content
+
+    service = parser["Service"]
+    assert service.get("Type") == "oneshot"
+    assert service.get("ExecStart") == "/usr/bin/systemd-sysupdate reboot"
+
+    install = parser["Install"]
+    assert install.get("Also") == "systemd-sysupdate-reboot.timer"
+
+
+def test_reboot_timer_contract():
+    parser = load_ini(REBOOT_TIMER)
+    assert parser.has_section("Unit")
+    assert parser.has_section("Timer")
+    assert parser.has_section("Install")
+
+    unit = parser["Unit"]
+    assert unit.get("ConditionVirtualization") == "!container"
+
+    timer = parser["Timer"]
+    # Maintenance window scheduling
+    assert timer.get("OnCalendar") == "*-*-* 04:10:00"
+    assert timer.get("RandomizedDelaySec") == "30min"
+    assert timer.get("Persistent") == "true"
+
+    install = parser["Install"]
+    assert install.get("WantedBy") == "timers.target"
+
+
+def test_reboot_preset_enables_timer_by_default():
+    content = REBOOT_PRESET.read_text()
+    assert "enable systemd-sysupdate-reboot.timer" in content
+
+
+def test_kured_hook_preserved_in_sysupdate_service():
+    assert KURED_HOOK.is_file(), "kured-hook.conf is missing"
+    content = KURED_HOOK.read_text()
+    assert "touch /run/reboot-required" in content
+
+
+def test_reboot_coordination_packaged_in_elements():
+    # systemd unit files packaging
+    system_element = (ELEMENTS_DIR / "os-k0s-first-boot.bst").read_text()
+    assert "path: files/os/systemd/system" in system_element
+    assert "target: /usr/lib/systemd/system" in system_element
+
+    # systemd preset files packaging
+    preset_element = (ELEMENTS_DIR / "os-sshd-preset.bst").read_text()
+    assert "path: files/os/systemd/system-preset" in preset_element
+    assert "target: /usr/lib/systemd/system-preset" in preset_element
+
+    # kured hook packaging
+    kured_element = (ELEMENTS_DIR / "os-kured-hook.bst").read_text()
+    assert "path: files/os/systemd/systemd-sysupdate.service.d" in kured_element
+    assert "target: /usr/lib/systemd/system/systemd-sysupdate.service.d" in kured_element
+
+    # os-stack packaging
+    stack_content = OS_STACK.read_text()
+    assert "bluefin-server/os-k0s-first-boot.bst" in stack_content
+    assert "bluefin-server/os-sshd-preset.bst" in stack_content
+    assert "bluefin-server/os-kured-hook.bst" in stack_content