Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions Justfile
Original file line number Diff line number Diff line change
Expand Up @@ -490,7 +490,7 @@ install-vm:
}
trap cleanup INT TERM

until curl --silent --show-error --max-time 2 --output /dev/null http://127.0.0.1:8080/; do
until curl --silent --insecure --max-time 2 --output /dev/null https://127.0.0.1:8080/ || curl --silent --max-time 2 --output /dev/null http://127.0.0.1:8080/; do
if ! kill -0 "$QEMU_PID" 2>/dev/null; then
wait "$QEMU_PID"
exit 1
Expand All @@ -500,7 +500,7 @@ install-vm:

HOST_IP="$(ip -4 -o addr show scope global | awk '{print $4}' | cut -d/ -f1 | head -n1)"
echo "==> KubeStellar Console is ready!"
echo "==> Access URL (LAN): http://${HOST_IP:-localhost}:8080/"
echo "==> Access URL (Local): http://localhost:8080/"
xdg-open "http://${HOST_IP:-localhost}:8080/" || xdg-open http://localhost:8080/ || true
echo "==> Access URL (LAN): https://${HOST_IP:-localhost}:8080/"
echo "==> Access URL (Local): https://localhost:8080/"
xdg-open "https://${HOST_IP:-localhost}:8080/" || xdg-open https://localhost:8080/ || true
wait "$QEMU_PID"
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Bluefin Server is currently in **Alpha**:
- **systemd-native installer** — `systemd-sysinstall` provides the interactive terminal UI and `systemd-repart` handles partitioning and block-copy DDI placement.
- **Optional k0s as a `systemd-sysext`** so the base image stays minimal.

> **Remote diagnostics:** OpenSSH is installed for on-demand diagnostics, but is disabled by default via systemd presets. It can be started manually with `systemctl start sshd` when remote access is needed. See [`docs/skills/factory-integration.md`](docs/skills/factory-integration.md).
> **Remote diagnostics:** SSH is key-only for the provisioned `core` operator account and is enabled after successful key provisioning. See [`docs/skills/factory-integration.md`](docs/skills/factory-integration.md).

## Quick start

Expand Down
10 changes: 7 additions & 3 deletions docs/MVP_1_0_READINESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This audit tracks the gap between the current tree and a first public/usable MVP
2. **Signed release artifacts** — combined `SHA256SUMS` + detached GPG signature published to GitHub Releases.
3. **Automated boot verification** — at least one non-human test that proves the installer writes a bootable disk and the installed OS reaches a target.
4. **Functional update path** — host can pull the signed manifest and apply an OS update without manual intervention.
5. **Basic first-boot provisioning** — unattended way to set root credential and drop an SSH authorized key.
5. **Basic first-boot provisioning** — unattended `core` operator key-provisioning model for SSH authorized keys.
6. **Documented recovery** — A/B rollback or reinstall-from-media path for a failed update.

## Current state
Expand All @@ -22,7 +22,7 @@ This audit tracks the gap between the current tree and a first public/usable MVP
| Automated boot test | 🔄 | Phase B in progress for Alpha; `bluefin-server-boot-test` workflow running on lab cluster |
| A/B root rollback | ❌ | `50-root.transfer` names `root-a`/`root-b`, installer only creates `root-a` |
| Root immutability | ❌ | DDI boots read/write (`rw` on cmdline) |
| First-boot SSH keys | | Only root password credential path exists |
| First-boot SSH keys | | Implemented `tmpfiles.extra` path writing `/var/home/core/.ssh/authorized_keys` |

Competitor context: [gap-analysis-distros.md](skills/gap-analysis-distros.md)

Expand Down Expand Up @@ -53,14 +53,18 @@ Priority order. Each item depends on the ones above it.

- [ ] Add `root-b` to installer repart recipes and verify `systemd-sysupdate` stages into the inactive slot.
- [ ] Switch UKI cmdline from `rw` to `ro` and rely on `/var` for mutable state.
- [ ] Consume `systemd-creds` for SSH authorized keys and static network config.
- [ ] Consume `systemd-creds` for static network config.
- [ ] Add boot menu entry to select the previous slot after a failed update.

### Phase D: release discipline

- [ ] Tag `v1.0.0-MVP` once Phase B passes.
- [ ] Publish release notes: verified boot path, trust model, known gaps.

## Migration prerequisites

Users of `root / bluefin` must provision the `core` key credential before booting the hardened release, and recovery is offline ESP replacement of `/loader/credentials/tmpfiles.extra.cred`. There is no password or root fallback.

## Open decisions

- Does MVP 1.0 require A/B dual-slot rollback, or is a single-slot signed update with documented reinstall recovery acceptable for the first release?
Expand Down
4 changes: 2 additions & 2 deletions docs/skills/architecture-roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Roadmap for future Bluefin Server architecture work. Use when plann
metadata:
type: reference
status: stable
last_updated: "2026-09-07"
last_updated: "2026-09-09"
context7-sources:
- /systemd/systemd
---
Expand All @@ -25,7 +25,7 @@ Priorities are derived from [gap-analysis-distros.md](gap-analysis-distros.md).
| 1 | A/B dual-slot root partitions with matching ESP/UKI slots | Root fs only has slot A today; sysupdate already names slots A+B. |
| 2 | Mount `/usr` read-only and enforce the state model | DDI is currently booted `rw`; sysext-first design assumes immutable `/usr`. |
| 3 | Boot-time selection / automatic rollback of a failed update | No previous OS version is kept once a root update overwrites the slot. |
| 4 | Broader `systemd-creds` integration for SSH keys and network configuration | Only `root` password credential path is shipped in `os-creds-prov.bst`. |
| 4 | Broader `systemd-creds` integration for static network configuration | `core` SSH keys already arrive through the `tmpfiles.extra` credential; no credential path ships static network configuration. |
| 5 | TPM2-bound credential delivery at first boot | Documented but not wired into the installed OS image. |
| 6 | Native reboot coordination for non-Kubernetes and single-node hosts | Kured only covers Kubernetes nodes; no FleetLock/locksmith equivalent. |
| 7 | Staged rollout behavior for larger fleets | Future after items 1-3 are implemented. |
Expand Down
7 changes: 6 additions & 1 deletion docs/skills/factory-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,12 @@ The workloads the factory tests and ships live in other repositories or image pi

## SSH and Remote Diagnostics

> `sshd` is present in the OS image for on-demand diagnostics and bring-up troubleshooting, but is disabled by default via `disable sshd.service` in systemd presets. Operators can start it on-demand with `systemctl start sshd` or enable it when remote access is required. Root login is permitted with password and pubkey.
SSH access is key-only for the `core` operator account (`PermitRootLogin no`, `PasswordAuthentication no`). Root login and password authentication are completely disabled. `sshd` is enabled at boot and gated by `bluefin-core-access.service`, starting only after `core` authorization keys are provisioned via `tmpfiles.extra`. Operators connect as `core` and elevate with passwordless sudo:

```sh
ssh core@server.example
sudo -i
```

## When to Use

Expand Down
10 changes: 5 additions & 5 deletions docs/skills/gap-analysis-distros.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from
| **Philosophy** | Systemd-native, minimal, image-based server OS appliance; base DDI includes bash for login and bring-up while heavy developer/debug tools live in sysexts or system containers; intended to run container workloads and Kubernetes via optional sysexts. Sources: [AGENTS.md](../../AGENTS.md), [factory-integration.md](factory-integration.md). |
| **State model** | Target OS DDI is an XFS filesystem image. A separate persistent `/var` partition is created by the installer. There is no second root slot provisioned today, and the UKI cmdline currently uses `rw`, so the root is not mounted read-only at runtime. Sources: [bluefin-server-ddi.bst](../../elements/oci/bluefin-server-ddi.bst), [20-root-a.conf](../../files/installer/repart.d/20-root-a.conf), [bluefin-server-installer.bst](../../elements/oci/bluefin-server-installer.bst). |
| **Updates** | `systemd-sysupdate` reads root/UKI transfers from `files/os/sysupdate.d/` and the optional k0s transfer from the `k0s` component directory. Assets are published to GitHub Releases, and the combined `SHA256SUMS` manifest is signed in CI with a GPG key. `Verify=yes` is the default. Sources: [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md), [50-root.transfer](../../files/os/sysupdate.d/50-root.transfer), [60-uki.transfer](../../files/os/sysupdate.d/60-uki.transfer), [70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer), also `systemd-sysupdate(8)`. |
| **Provisioning** | The installer is an offline `systemd-sysinstall` image that embeds the DDI as a data partition. First-boot configuration is intended to be delivered via `systemd-creds` through the ESP or hypervisor metadata. Today only `passwd.hashed-password.root` is consumed via `systemd-sysusers.d`; the documented `tmpfiles.extra` path for SSH keys and similar files is not implemented. Sources: [bluefin-server-installer.bst](../../elements/oci/bluefin-server-installer.bst), [10-root-creds.conf](../../files/os/sysusers.d/10-root-creds.conf), [os-creds-prov.bst](../../elements/bluefin-server/os-creds-prov.bst), [systemd-creds(1)](https://www.freedesktop.org/software/systemd/man/latest/systemd-creds.html). |
| **Provisioning** | The installer is an offline `systemd-sysinstall` image that embeds the DDI as a data partition. First-boot configuration is delivered via `systemd-creds` through the ESP or hypervisor metadata. `core` keys are written by the `tmpfiles.extra` credential into persistent `/var/home/core/.ssh/authorized_keys`. Sources: [bluefin-server-installer.bst](../../elements/oci/bluefin-server-installer.bst), [10-core-user.conf](../../files/os/sysusers.d/10-core-user.conf), [os-creds-prov.bst](../../elements/bluefin-server/os-creds-prov.bst), [systemd-creds(1)](https://www.freedesktop.org/software/systemd/man/latest/systemd-creds.html). |
| **Customization** | Adds software through `systemd-sysext` (overlay `/usr`) and `systemd-confext` (overlay `/etc`) images. The base OS `os-release` advertises `ID=flatcar` and a matching `VERSION_ID` so pre-built Flatcar Bakery extensions load. k0s is shipped as a separately built, optionally enabled sysext. Sources: [systemd-sysext-extensions.md](systemd-sysext-extensions.md), [k0s-sysext.md](k0s-sysext.md), [os-release-flatcar.bst](../../elements/bluefin-server/os-release-flatcar.bst), [systemd-sysext(8)](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html). |
| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStartPost` that touches `/run/reboot-required`. Rolling reboots across Kubernetes nodes rely on Kured reading that file. Sources: [os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst), [kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf), [Kured project](https://github.com/weaveworks/kured). |

Expand All @@ -112,8 +112,7 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from

### Provisioning

- **Gap:** The documented `systemd-creds` first-boot provisioning for SSH keys (`tmpfiles.extra`) is not present in the build (`os-creds-prov.bst` only ships `sysusers.d`).
Operators can only pre-seed the root password today.
- **Gap:** `core` keys are written by the `tmpfiles.extra` credential into persistent `/var/home/core/.ssh/authorized_keys`, but broader first-boot credential provisioning (such as network configuration or multiple operator identities) is not yet implemented.
- **Gap:** `systemd-firstboot` is masked in the installer environment, so interactive first-boot questions are skipped; any further user/network/timezone configuration must be supplied through credentials that the build does not yet consume.
- **Gap:** TPM2 sealing for credentials is documented in `docs/skills/tpm2-credential-sealing.md`, but there is no evidence in the OS build that sealed credentials are generated, shipped, or decrypted automatically during first boot.

Expand All @@ -134,7 +133,7 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from

1. **A/B dual-slot rollback is not wired end-to-end.** The `sysupdate` transfer names `root-a`/`root-b`, but the installer only provisions `root-a`, so Bluefin cannot atomically stage and roll back a new root image today.
2. **Root filesystem immutability is not enforced.** The DDI is built and booted read-write; the intended read-only `/usr` + overlay model depends entirely on optional sysext behavior rather than runtime policy.
3. **First-boot credential provisioning is incomplete.** Only the `root` password credential path is shipped; SSH keys, network configuration, and other `systemd-creds`-based provisioning remain documented but not implemented.
3. **First-boot credential provisioning is incomplete.** `core` keys are written by the `tmpfiles.extra` credential into persistent `/var/home/core/.ssh/authorized_keys`, but network configuration and other `systemd-creds`-based provisioning remain documented but not implemented.

These gaps drive the priorities in [architecture-roadmap.md](architecture-roadmap.md).

Expand Down Expand Up @@ -193,5 +192,6 @@ These gaps drive the priorities in [architecture-roadmap.md](architecture-roadma
- [files/os/sysupdate.d/50-root.transfer](../../files/os/sysupdate.d/50-root.transfer)
- [files/os/sysupdate.d/60-uki.transfer](../../files/os/sysupdate.d/60-uki.transfer)
- [files/os/sysupdate.k0s.d/70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer)
- [files/os/sysusers.d/10-root-creds.conf](../../files/os/sysusers.d/10-root-creds.conf)
- [files/os/sysusers.d/10-core-user.conf](../../files/os/sysusers.d/10-core-user.conf)
- [files/os/tmpfiles.d/10-core-home.conf](../../files/os/tmpfiles.d/10-core-home.conf)
- [files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf)
59 changes: 44 additions & 15 deletions docs/skills/tpm2-credential-sealing.md
Original file line number Diff line number Diff line change
@@ -1,25 +1,25 @@
---
name: tpm2-credential-sealing
description: Securing provisioning credentials (such as hashed root passwords or SSH keys) with TPM2 sealing via systemd-creds.
description: Securing provisioning credentials (such as operator SSH keys via tmpfiles.extra) with TPM2 sealing via systemd-creds.
metadata:
type: how-to
status: stable
last_updated: "2026-09-07"
last_updated: "2026-09-09"
context7-sources:
- /systemd/systemd
---
# TPM2 Credential Sealing

To secure sensitive provisioning credentials (such as hashed root passwords or
SSH keys) against physical tampering or unauthorized extraction, bind them to
the TPM2 and the UKI boot state using `systemd-creds`.
To secure sensitive provisioning credentials (such as operator SSH keys via
`tmpfiles.extra`) against offline tampering, bind them to the TPM2 and the UKI
boot state using `systemd-creds`.

## Current scope

This skill is the canonical home for sealed credentials such as hashed root
passwords and similar provisioning secrets. Broader `systemd-creds` integration
for SSH keys and network configuration is a future roadmap item in
[architecture-roadmap.md](architecture-roadmap.md).
This skill is the canonical home for sealed credentials, specifically the
`tmpfiles.extra` credential used to provision SSH access for the `core` operator
account. Broader `systemd-creds` integration for network configuration is a
future roadmap item in [architecture-roadmap.md](architecture-roadmap.md).

## Verify TPM2 device availability

Expand All @@ -32,20 +32,33 @@ not running in a container.

## Encrypt and seal a credential

The `tmpfiles.extra` credential payload establishes directory ownership and
writes the authorized SSH keys for the `core` operator account:

```text
d /var/home/core 0700 core core -
d /var/home/core/.ssh 0700 core core -
f~ /var/home/core/.ssh/authorized_keys 0600 core core - c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5UR...
```

The base64 data in the `f~` line is an SSH public key, not a secret. Sealing
the credential ensures that unauthorized keys cannot be injected into the
machine offline when TPM2 protection is active.

Seal the credential against PCR 7 (Secure Boot state) and PCR 11 (Unified Kernel
Image state) on the TPM2 chip:

```bash
systemd-creds encrypt \
--name=passwd.hashed-password.root \
--name=tmpfiles.extra \
--with-key=tpm2 \
--tpm2-pcrs=7+11 \
/path/to/plaintext_password_hash.txt \
/path/to/plaintext_tmpfiles_extra.txt \
/path/to/secured_credential.cred
```

- `--name=` must match the credential name the consumer expects (for example,
`passwd.hashed-password.root` is read by `systemd-sysusers`).
`tmpfiles.extra` is read by `systemd-tmpfiles`).
- `--with-key=tpm2` forces a TPM2-bound credential. The default `auto` also uses
the host key if `/var/lib/systemd/` is on persistent media; omit the switch
if you want both bindings.
Expand All @@ -55,19 +68,35 @@ systemd-creds encrypt \
## Provide the encrypted credential to the host

Place the output `.cred` file in the ESP credential directory or pass it via a
container/hypervisor mechanism:
container/hypervisor mechanism. The credential file is
`/loader/credentials/tmpfiles.extra.cred` on the target ESP:

```bash
# ESP delivery
mkdir -p /loader/credentials/
cp /path/to/secured_credential.cred /loader/credentials/passwd.hashed-password.root.cred
cp /path/to/secured_credential.cred /loader/credentials/tmpfiles.extra.cred

# Or via a container/hypervisor argument
--set-credential=passwd.hashed-password.root:/path/to/secured_credential.cred
--set-credential=tmpfiles.extra:/path/to/secured_credential.cred
```

## Offline recovery sequence

There is no fallback password or root SSH bypass. If the `core` key is lost or
unusable, SSH fails closed. The supported offline recovery sequence requires
physical access or equivalent hypervisor access:

1. Shut down the target host.
2. Mount the target ESP from a trusted machine.
3. Replace `/loader/credentials/tmpfiles.extra.cred` on the ESP with a
credential that writes the replacement `core` public key.
4. Boot the host. `systemd-tmpfiles` applies the credential, `bluefin-core-access`
verifies key readiness, and `sshd` starts.
5. Authenticate as `core` (`ssh core@server.example`) and elevate using `sudo -i`.

## See also

- [CONTEXT.md](../../CONTEXT.md) — canonical project domain glossary.
- [factory-integration.md](factory-integration.md) — operator login and remote diagnostics.
- `systemd-creds(1)`
- `systemd.system-credentials(7)`
Loading