diff --git a/docs/skills/architecture-roadmap.md b/docs/skills/architecture-roadmap.md index afa8eb0..ff278a2 100644 --- a/docs/skills/architecture-roadmap.md +++ b/docs/skills/architecture-roadmap.md @@ -27,7 +27,7 @@ Priorities are derived from [gap-analysis-distros.md](gap-analysis-distros.md). | 3 | Boot-time selection / automatic rollback of a failed update | No previous OS version is kept once a root update overwrites the slot. | | 4 | Broader `systemd-creds` integration for SSH keys and network configuration | Only `root` password credential path is shipped in `os-creds-prov.bst`. | | 5 | TPM2-bound credential delivery at first boot | Documented but not wired into the installed OS image. | -| 6 | Native reboot coordination for non-Kubernetes and single-node hosts | Kured only covers Kubernetes nodes; no FleetLock/locksmith equivalent. | +| 6 | Distributed reboot lock coordination for multi-node non-Kubernetes clusters | Single-node / maintenance window / lock-file reboot coordination is present via `systemd-sysupdate-reboot`; cluster-wide FleetLock/locksmith HTTP protocol is not implemented. | | 7 | Staged rollout behavior for larger fleets | Future after items 1-3 are implemented. | ## Status notes diff --git a/docs/skills/gap-analysis-distros.md b/docs/skills/gap-analysis-distros.md index 12b4413..558a2b2 100644 --- a/docs/skills/gap-analysis-distros.md +++ b/docs/skills/gap-analysis-distros.md @@ -99,7 +99,7 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from | **Updates** | `systemd-sysupdate` reads root/UKI transfers from `files/os/sysupdate.d/` and the optional k0s transfer from the `k0s` component directory. Assets are published to GitHub Releases, and the combined `SHA256SUMS` manifest is signed in CI with a GPG key. `Verify=yes` is the default. Sources: [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md), [50-root.transfer](../../files/os/sysupdate.d/50-root.transfer), [60-uki.transfer](../../files/os/sysupdate.d/60-uki.transfer), [70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer), also `systemd-sysupdate(8)`. | | **Provisioning** | The installer is an offline `systemd-sysinstall` image that embeds the DDI as a data partition. First-boot configuration is intended to be delivered via `systemd-creds` through the ESP or hypervisor metadata. Today only `passwd.hashed-password.root` is consumed via `systemd-sysusers.d`; the documented `tmpfiles.extra` path for SSH keys and similar files is not implemented. Sources: [bluefin-server-installer.bst](../../elements/oci/bluefin-server-installer.bst), [10-root-creds.conf](../../files/os/sysusers.d/10-root-creds.conf), [os-creds-prov.bst](../../elements/bluefin-server/os-creds-prov.bst), [systemd-creds(1)](https://www.freedesktop.org/software/systemd/man/latest/systemd-creds.html). | | **Customization** | Adds software through `systemd-sysext` (overlay `/usr`) and `systemd-confext` (overlay `/etc`) images. The base OS `os-release` advertises `ID=flatcar` and a matching `VERSION_ID` so pre-built Flatcar Bakery extensions load. k0s is shipped as a separately built, optionally enabled sysext. Sources: [systemd-sysext-extensions.md](systemd-sysext-extensions.md), [k0s-sysext.md](k0s-sysext.md), [os-release-flatcar.bst](../../elements/bluefin-server/os-release-flatcar.bst), [systemd-sysext(8)](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html). | -| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStartPost` that touches `/run/reboot-required`. Rolling reboots across Kubernetes nodes rely on Kured reading that file. Sources: [os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst), [kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf), [Kured project](https://github.com/weaveworks/kured). | +| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStartPost` that touches `/run/reboot-required` for Kured. For non-Kubernetes and single-node hosts, `systemd-sysupdate-reboot.timer` and `systemd-sysupdate-reboot.service` coordinate automated reboots using systemd primitives, supporting maintenance window timers and lock inhibition (`/run/reboot-lock`, `/etc/reboot-lock`, `/run/kured-active`). Sources: [os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst), [kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf), [systemd-sysupdate-reboot.service](../../files/os/systemd/system/systemd-sysupdate-reboot.service), [systemd-sysupdate-reboot.timer](../../files/os/systemd/system/systemd-sysupdate-reboot.timer), [Kured project](https://github.com/weaveworks/kured). | ## 4. Factual Gaps @@ -120,7 +120,6 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from ### Update delivery - **Gap:** The root transfer uses `Type=partition Path=auto`, which requires `systemd-sysupdate` to discover a matching GPT partition label (`bluefin-server-root-a`/`root-b`). This is correct, but without a `root-b` partition the transfer effectively overwrites the running root in place. -- **Gap:** `systemd-sysupdate-reboot.service`/`systemd-sysupdate-reboot.timer` are not enabled or configured; the only reboot signal today is the Kured hook. ### Customization @@ -128,7 +127,8 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from ### Reboot coordination -- **Gap:** Kured coordinates Kubernetes node reboots but requires Kubernetes to be running. There is no equivalent for single-node or non-Kubernetes Bluefin hosts, and there is no built-in cluster lock manager similar to Zincati's FleetLock or Flatcar's `locksmithd`/`etcd-lock`. +- **Status:** Bluefin Server provides `systemd-sysupdate-reboot.timer` and `systemd-sysupdate-reboot.service` with maintenance window scheduling and lock-based inhibition (`/run/reboot-lock`, `/etc/reboot-lock`, `/run/kured-active`) for single-node and non-Kubernetes hosts, alongside the Kured hook for Kubernetes clusters. +- **Gap:** While single-node and lock-file inhibition is supported via systemd primitives, there is no built-in cluster-wide HTTP lock manager client similar to Zincati's FleetLock or Flatcar's `locksmithd`/`etcd-lock`. ## 5. Summary of Biggest Gaps @@ -195,3 +195,6 @@ These gaps drive the priorities in [architecture-roadmap.md](architecture-roadma - [files/os/sysupdate.k0s.d/70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer) - [files/os/sysusers.d/10-root-creds.conf](../../files/os/sysusers.d/10-root-creds.conf) - [files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf) +- [files/os/systemd/system/systemd-sysupdate-reboot.service](../../files/os/systemd/system/systemd-sysupdate-reboot.service) +- [files/os/systemd/system/systemd-sysupdate-reboot.timer](../../files/os/systemd/system/systemd-sysupdate-reboot.timer) +- [files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset](../../files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset) diff --git a/files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset b/files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset new file mode 100644 index 0000000..051bc45 --- /dev/null +++ b/files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset @@ -0,0 +1,2 @@ +# Enable automatic reboot timer for system updates on non-Kubernetes and single-node hosts +enable systemd-sysupdate-reboot.timer diff --git a/files/os/systemd/system/systemd-sysupdate-reboot.service b/files/os/systemd/system/systemd-sysupdate-reboot.service new file mode 100644 index 0000000..7fac775 --- /dev/null +++ b/files/os/systemd/system/systemd-sysupdate-reboot.service @@ -0,0 +1,14 @@ +[Unit] +Description=Reboot Automatically After System Update +Documentation=man:systemd-sysupdate-reboot.service(8) +ConditionVirtualization=!container +ConditionPathExists=!/run/reboot-lock +ConditionPathExists=!/etc/reboot-lock +ConditionPathExists=!/run/kured-active + +[Service] +Type=oneshot +ExecStart=/usr/bin/systemd-sysupdate reboot + +[Install] +Also=systemd-sysupdate-reboot.timer diff --git a/files/os/systemd/system/systemd-sysupdate-reboot.timer b/files/os/systemd/system/systemd-sysupdate-reboot.timer new file mode 100644 index 0000000..120a20f --- /dev/null +++ b/files/os/systemd/system/systemd-sysupdate-reboot.timer @@ -0,0 +1,12 @@ +[Unit] +Description=Reboot Automatically After System Update +Documentation=man:systemd-sysupdate-reboot.service(8) +ConditionVirtualization=!container + +[Timer] +OnCalendar=*-*-* 04:10:00 +RandomizedDelaySec=30min +Persistent=true + +[Install] +WantedBy=timers.target diff --git a/tests/unit/test_reboot_coordination.py b/tests/unit/test_reboot_coordination.py new file mode 100644 index 0000000..ecdc896 --- /dev/null +++ b/tests/unit/test_reboot_coordination.py @@ -0,0 +1,113 @@ +"""Unit tests for systemd-sysupdate reboot coordination. + +Asserts contracts for systemd-sysupdate-reboot service, timer, presets, +and drop-in coordination strategies (immediate, maintenance window, lock-based) +while ensuring Kured reboot paths are preserved without collisions. +""" + +import configparser +from pathlib import Path +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[2] +SYSTEMD_SYSTEM_DIR = REPO_ROOT / "files" / "os" / "systemd" / "system" +SYSTEMD_PRESET_DIR = REPO_ROOT / "files" / "os" / "systemd" / "system-preset" +SYSUPDATE_DROPIN_DIR = ( + REPO_ROOT / "files" / "os" / "systemd" / "systemd-sysupdate.service.d" +) +ELEMENTS_DIR = REPO_ROOT / "elements" / "bluefin-server" +OS_STACK = ELEMENTS_DIR / "os-stack.bst" + +REBOOT_SERVICE = SYSTEMD_SYSTEM_DIR / "systemd-sysupdate-reboot.service" +REBOOT_TIMER = SYSTEMD_SYSTEM_DIR / "systemd-sysupdate-reboot.timer" +REBOOT_PRESET = SYSTEMD_PRESET_DIR / "zz-enable-sysupdate-reboot.preset" +KURED_HOOK = SYSUPDATE_DROPIN_DIR / "kured-hook.conf" + + +def load_ini(path: Path) -> configparser.ConfigParser: + parser = configparser.ConfigParser(strict=False) + parser.optionxform = str + parser.read_string(path.read_text()) + return parser + + +def test_reboot_service_and_timer_exist(): + assert REBOOT_SERVICE.is_file(), "systemd-sysupdate-reboot.service is missing" + assert REBOOT_TIMER.is_file(), "systemd-sysupdate-reboot.timer is missing" + assert REBOOT_PRESET.is_file(), "zz-enable-sysupdate-reboot.preset is missing" + + +def test_reboot_service_contract(): + parser = load_ini(REBOOT_SERVICE) + assert parser.has_section("Unit") + assert parser.has_section("Service") + assert parser.has_section("Install") + + unit = parser["Unit"] + assert unit.get("ConditionVirtualization") == "!container" + + # Verify lock-based coordination and kured coordination guards + content = REBOOT_SERVICE.read_text() + assert "ConditionPathExists=!/run/reboot-lock" in content + assert "ConditionPathExists=!/etc/reboot-lock" in content + assert "ConditionPathExists=!/run/kured-active" in content + + service = parser["Service"] + assert service.get("Type") == "oneshot" + assert service.get("ExecStart") == "/usr/bin/systemd-sysupdate reboot" + + install = parser["Install"] + assert install.get("Also") == "systemd-sysupdate-reboot.timer" + + +def test_reboot_timer_contract(): + parser = load_ini(REBOOT_TIMER) + assert parser.has_section("Unit") + assert parser.has_section("Timer") + assert parser.has_section("Install") + + unit = parser["Unit"] + assert unit.get("ConditionVirtualization") == "!container" + + timer = parser["Timer"] + # Maintenance window scheduling + assert timer.get("OnCalendar") == "*-*-* 04:10:00" + assert timer.get("RandomizedDelaySec") == "30min" + assert timer.get("Persistent") == "true" + + install = parser["Install"] + assert install.get("WantedBy") == "timers.target" + + +def test_reboot_preset_enables_timer_by_default(): + content = REBOOT_PRESET.read_text() + assert "enable systemd-sysupdate-reboot.timer" in content + + +def test_kured_hook_preserved_in_sysupdate_service(): + assert KURED_HOOK.is_file(), "kured-hook.conf is missing" + content = KURED_HOOK.read_text() + assert "touch /run/reboot-required" in content + + +def test_reboot_coordination_packaged_in_elements(): + # systemd unit files packaging + system_element = (ELEMENTS_DIR / "os-k0s-first-boot.bst").read_text() + assert "path: files/os/systemd/system" in system_element + assert "target: /usr/lib/systemd/system" in system_element + + # systemd preset files packaging + preset_element = (ELEMENTS_DIR / "os-sshd-preset.bst").read_text() + assert "path: files/os/systemd/system-preset" in preset_element + assert "target: /usr/lib/systemd/system-preset" in preset_element + + # kured hook packaging + kured_element = (ELEMENTS_DIR / "os-kured-hook.bst").read_text() + assert "path: files/os/systemd/systemd-sysupdate.service.d" in kured_element + assert "target: /usr/lib/systemd/system/systemd-sysupdate.service.d" in kured_element + + # os-stack packaging + stack_content = OS_STACK.read_text() + assert "bluefin-server/os-k0s-first-boot.bst" in stack_content + assert "bluefin-server/os-sshd-preset.bst" in stack_content + assert "bluefin-server/os-kured-hook.bst" in stack_content