Skip to content

feat(bundle): Ship hvi, the overlay-block root and urunit as the initrd - #24

Draft
ananos wants to merge 9 commits into
mainfrom
feat/single-binary-overlay-block
Draft

ananos wants to merge 9 commits into
mainfrom
feat/single-binary-overlay-block

Conversation

@ananos

@ananos ananos commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

This branch puts hvi in the bundle and moves urunc to the stack that boots a container on it: one urunit binary as the whole initrd, an overlay root on two disks, and stop signals handed to the guest init. cloud-hypervisor stays in the bundle, as the fallback and as the A/B control.

The bundle names no default monitor. urunc boots whatever com.urunc.unikernel.hypervisor asks for, and brig sets that annotation. So a brig that knows hvi on Linux (brig-sh/brig feat/linux-rootfs-mode) boots hvi from this bundle, and an older brig keeps getting cloud-hypervisor. The same holds for com.urunc.unikernel.rootfsMode=overlay-block. The two land in that order: this bundle first, then the brig change with the install.sh pin.

Changes

  • hvi, built from source. brig-sh/hvi-vmm publishes no Linux binary, so the build compiles it at a pinned commit (24dc2c5, main), static against musl in a rust:1.95.0 container, and installs it as bin/hvi with a [monitors.hvi] section. MONITORS lists it first. The pin is main 24dc2c5, which carries the vsock half-close fix (fix(vsock): End each direction of a connection on its own brig-sh/hvi-vmm#266); that one changes the Linux binary, so the branch follows it.
  • urunc at ff1063a, feat/linux-overlay-block: feat/unchanged_containers with the exec fixes, the Linux hvi backend with exec over its agent socket, urunit's single binary in the initrd, rootfsMode=overlay-block (a read-only lower disk and a writable upper, both with serials), stop signals to the guest init, and urunit.conf as the NUL-terminated records of fix(unikernels): use NUL separation for urunit config values urunc-dev/urunc#1111. It sits on urunc main 6545f17.
  • urunit from urunc's pin. The build reads packaging/container-initrd/urunit.pin from the urunc checkout (today de324cf, stack/9-nul-without-magic), so the initrd always matches the shim that talks to it. URUNIT_REPO/URUNIT_REF override it for one build; a urunc commit without the file is refused. busybox and container-init leave the initrd: it is /urunit with /init and /vz-init linked to it.
  • Pin test and CI check follow: the stub urunc checkout plants a urunit.pin, and CI compares URUNIT_REF in pins.env with the pin at the recorded urunc commit.
  • Docs say who picks the monitor, count hvi among the four things the build compiles, and close the open question on where hvi comes from.

Tested (amd64, nofire-cerno)

Built with this branch at e07701d (--rootless): pins.env records urunc ff1063a, urunit de324cf, hvi 7766307, all PINNED=true; bin/hvi --version is hvi 0.1.0 (core 0.1.0), static-pie musl; the initrd is urunit (136 KB) plus the two links.

User install over an rc17 install, then ubuntu:24.04 through brig-ctl run with brig's boot annotations:

Boot Result
hvi, virtio-fs root boots (guest 6.18.55), exec prints both lines of a two-line command
cloud-hypervisor, virtio-fs root same
hvi, rootfsMode=overlay-block --disk ro lower (disk0) + rw upper (disk1), / is overlay, a file written to the guest root is still there after nerdctl stop and start

No monitor or virtiofsd left after rm. tests/source-pins.sh, the other tests/*.sh, sh -n and shellcheck are clean.

Earlier runs of the same pins (21 and 23 October, reduced then full size) on nuc8, jetson6 and rpi5-11: hvi at parity with cloud-hypervisor on every stress row, 0 truncated of 2000 concurrent brig sh, 0 of 20 overlay-block writes lost, 0 of 500 execs failed.

nuc8, from the CI artifacts of e07701d (fresh rootless install, then the full s6 gate): stress 60/60 on hvi, cloud-hypervisor, and both with the overlay-block root; conc3 2000/2000 on each monitor with 0 truncated and 0 ^@; nul-ab 500/500; persist 0 of 20 lost; exec 0 of 500 failed; flock across sandboxes 9 and 14 of 1000 (virtiofsd, same on both). Then a system install from the plain artifact, one hvi boot as root through /usr/local/bin/brig, and an uninstall that leaves no /var/lib/brig, launcher or /etc/urunc; the user install was intact afterwards and came back after a reboot of the host.

The pin bump to brig v0.4.1, hvi 24dc2c5 and assets 0.1.10 came after that gate; the hvi move changes the Linux binary (vsock), so nuc8 was re-gated from the CI artifact of 8db81ee: in-place user upgrade (pins.env hvi 24dc2c5, brig v0.4.1, assets 0.1.10), the three smoke boots, a 4 MB guest-to-host stream complete on both monitors, conc3 2000/2000 with 0 truncated and 0 ^@ on hvi and on cloud-hypervisor, persist 0 of 20 lost and exec 0 of 500 failed on hvi, no leftovers. The urunc pin will move once the stack is rebased on current main (7 commits behind, CI and deps only).

Release sequence

  1. Merge this and cut v0.1.0-rc18.
  2. brig: feat/linux-rootfs-mode plus the install.sh RUNTIME_VERSION bump to that tag, in one PR.

brig is moving its Linux generic boot from cloud-hypervisor to hvi. Ship
hvi in the bundle, with its own [monitors.hvi] section in urunc.toml, and
take the urunc that knows the monitor.

- hvi has no Linux release, so the build checks out brig-sh/hvi-vmm at a
  pinned commit and builds it static against musl in rust:1.95.0, the
  toolchain hvi-vmm pins. HVI_REF and HVI_BRANCH work as URUNC_REF and
  URUNC_BRANCH do, and pins.env records the commit.
- urunc moves to feat/hvi-linux-drop-in at a1e7959: the exec-fixes
  branch plus the hvi monitor, with exec over hvi's agent socket.
- cloud-hypervisor stays in the bundle, so a host can still name it.

hvi presents its devices over virtio-mmio. The hull-assets kernel already
builds VIRTIO_MMIO and VIRTIO_MMIO_CMDLINE_DEVICES in for hvi on macOS, so
the same kernel boots on both monitors.

tests/source-pins.sh and the CI record check cover the hvi pin.

Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
The hvi commit, a1e7959, is now on feat/unchanged_containers-exec-fixes,
the branch the bundle took its urunc from before. Name that branch again.
The pinned commit is the same.

Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
The bundle built its initrd from four pieces: urunc's container-init
script, a BusyBox, urunit from its own pin (71bfdee, urunit_agent) and
a urunit-agent built from the urunc checkout. urunc now boots one
static urunit for all of that, and pins the urunit commit it speaks to
in packaging/container-initrd/urunit.pin.

- urunc: feat/linux-overlay-block at 074ab76. On top of the exec-fixes
  branch it carries urunit's single binary as the initrd, the
  overlay-block root (com.urunc.unikernel.rootfsMode, which brig sets
  from BRIG_ROOTFS_MODE), read-only disks with serials and one
  virtio-fs export per volume on hvi and cloud-hypervisor, and stop
  signals handed to the guest's init.
- urunit: read from urunc's urunit.pin after the urunc checkout, so the
  initrd always matches the shim. URUNIT_REPO and URUNIT_REF still
  override it. The pin is 0b3157d, NOFireAI/urunit stack/7-stop-signals.
- initrd: the default single-binary layout, /urunit with /init and
  /vz-init linked to it. The BusyBox extraction is gone with it.
- hvi: 3c8b8fd, brig-sh/hvi-vmm fix/seccomp-glibc-arena. On top of main
  590c45d (one --disk per disk with serial=, read-only disks, an --fs
  per export) it keeps glibc's malloc off the files it would open under
  seccomp. Without it hvi died with SIGSYS on the first agent
  connection of a guest with two or more virtio-fs exports.

README, DESIGN and docs/variants.md describe the new initrd and pins.

Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
The overlay-block urunc stack is now rebased onto urunc main as of
6545f17, which moves the shim to containerd v2 and serves the task API
v3. On top of it, 7cf6db0 waits for the guest to power off when the
agent ends the stop signal's session, where urunc used to signal the
monitor at once and could lose the guest's last root writes. hvi main as
of e7d9adf fails a glibc build's opens under seccomp with EPERM
(brig-sh/hvi-vmm#311), so the bundle takes main and drops the
fix/seccomp-glibc-arena branch. The bundle builds hvi against musl,
which that bug never reached.

- urunc: 074ab76 -> 7cf6db0 (feat/linux-overlay-block, rebased, plus the
  stop fix).
- hvi: 3c8b8fd (fix/seccomp-glibc-arena) -> e7d9adf (main).
- urunit: unchanged, 0b3157d from urunc's urunit.pin.

Tested on nofire (rootless brig, containerd v2.3.5) with the binaries,
kernel and initrd of a bundle built from this commit: rootfsMode=
overlay-block on hvi and cloud-hypervisor 42/42 twice, stop signals in
both root modes 16/16.

Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
urunc ff1063a pins urunit de324cf and writes urunit.conf without the
URUNIT1 record, in the format of urunc-dev/urunc#1111 and
nubificus/urunit#17. The bundle takes urunit from that pin, so the
initrd moves with it:

- e5c0acd: the guest syncs before it stops the agent, so a stop no longer
  loses the application's last writes to an overlay-block root, even
  with a urunc that signals the monitor at once.
- de324cf: urunit reads NUL-terminated records with or without URUNIT1,
  and the legacy lines.

- urunc: 7cf6db0 -> ff1063a (feat/linux-overlay-block).
- urunit: 0b3157d -> de324cf (stack/9-nul-without-magic), from the pin.
- hvi: unchanged, e7d9adf.

Tested on nofire (rootless brig, containerd v2.3.5) with the binaries,
kernel and initrd of a bundle built from this commit: rootfsMode=
overlay-block on hvi and cloud-hypervisor 42/42, stop signals 16/16, an
environment value with new lines and a UEE line read verbatim on both
monitors and root modes, and 0 of 20 stops losing the last root write on
hvi.

Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
The build no longer carries a URUNIT_REF_DEFAULT: it takes urunit from the
packaging/container-initrd/urunit.pin of the urunc commit it pins, so the
initrd always matches the shim. The pin test and the CI pins check still
looked for the removed variable and failed on their first line.

The test's stub git now plants a urunit.pin when it checks out the urunc
source, and the test asserts urunit is fetched at that commit, that
URUNIT_REF overrides it for one build, and that a urunc commit without the
file is refused. CI reads the pin from the recorded urunc commit on GitHub
and compares it with URUNIT_REF in pins.env.

Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
hvi moves from e7d9adf to main 7766307, the commit hull pins. The four
commits between them touch the built-in virtio-fs server and HVF, which
the Linux build does not compile (exports go through virtiofsd), so the
Linux binary is unchanged; the pin moves so one hvi commit serves both
platforms.

The README said the bundle makes hvi the default monitor. It does not:
urunc boots the monitor the com.urunc.unikernel.hypervisor annotation
names and brig sets that annotation, so the switch is brig's, and an older
brig keeps getting cloud-hypervisor from this bundle. The build section
now says so, counts hvi among the things the build compiles, and closes
the open question on where hvi comes from.

Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
Three pins had moved under the branch. brig v0.4.1 is the release the
installer already pins, so the bundle carries the same brig. hvi main
24dc2c5 adds the vsock fix that ends each direction of a connection on
its own (brig-sh/hvi-vmm#266); unlike the earlier macOS-only commits this
one changes the Linux binary, which is why the pin follows it. hull-assets
0.1.10 carries the same 6.18.55 kernel as 0.1.9; its change is the initrd
agent, which this bundle does not take, so the move only keeps the bundle
on the current tag.

Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
The workflows named each action by a major tag, which moves under them.
Each is now pinned by the commit its latest release tag names:

| Action | From | To |
| --- | --- | --- |
| actions/checkout | v4 | v7.0.1 |
| actions/upload-artifact | v4 | v7.0.2 |
| actions/download-artifact | v4 | v8.0.2 |
| actions/attest-build-provenance | v2 | v4.2.2 |
| sigstore/cosign-installer | v3 | v4.1.2 |

The majors move the actions to Node 24 and change no input these
workflows pass; download-artifact v8 now fails on a digest mismatch.

cosign-installer v4 installs cosign v3 by default, whose sign-blob
writes the new bundle format. install.sh verifies the checksums'
signature as it is written today, so the release step names cosign
v2.5.3, the version v3 installed.

Signed-off-by: Anastassios Nanos <ananos@nofire.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant