Skip to content

Commit 44f67ac

Browse files
committed
refactor(vm): build kernel once on Linux, reuse kernel.c on macOS
Eliminate the krunvm/Fedora VM dependency from the macOS CI job by building the aarch64 Linux kernel only on the Linux ARM64 runner and exporting kernel.c as a CI artifact. The macOS job downloads kernel.c and compiles it into libkrunfw.dylib with Apple's cc, cutting macOS CI from ~45 min to ~5 min. Also fixes: - sudo not found in CI containers (use conditional SUDO) - Hardcoded ARCH=arm64 in build-libkrun.sh (now auto-detects) - Missing packages:read permission for GHCR pulls - brew tap ordering for Homebrew formula resolution Removes build-custom-libkrunfw.sh (no longer needed).
1 parent 16adfdc commit 44f67ac

8 files changed

Lines changed: 240 additions & 550 deletions

File tree

.github/workflows/release-vm-kernel.yml

Lines changed: 30 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,11 @@ name: Release VM Kernel
55
# "vm-dev" GitHub Release and consumed by release-vm-dev.yml when building the
66
# openshell-vm binary.
77
#
8+
# The Linux kernel is compiled once on aarch64 Linux. The resulting kernel.c
9+
# (a C source file containing the kernel as a byte array) is shared with the
10+
# macOS job, which only needs to compile it into a .dylib — no krunvm, no
11+
# Fedora VM, no kernel rebuild. This cuts macOS CI from ~45 min to ~5 min.
12+
#
813
# This workflow runs on-demand (or when kernel config / pins change). It is
914
# intentionally decoupled from the per-commit VM binary build because the
1015
# kernel rarely changes and takes 15-45 minutes to compile.
@@ -27,7 +32,7 @@ defaults:
2732

2833
jobs:
2934
# ---------------------------------------------------------------------------
30-
# Linux ARM64 — native kernel + libkrun build
35+
# Linux ARM64 — native kernel + libkrun build (also exports kernel.c)
3136
# ---------------------------------------------------------------------------
3237
build-runtime-linux-arm64:
3338
name: Build Runtime (Linux ARM64)
@@ -57,13 +62,25 @@ jobs:
5762
--build-dir target/libkrun-build \
5863
--output artifacts/vm-runtime-linux-aarch64.tar.zst
5964
60-
- name: Upload artifact
65+
- name: Upload runtime artifact
6166
uses: actions/upload-artifact@v4
6267
with:
6368
name: vm-runtime-linux-arm64
6469
path: artifacts/vm-runtime-linux-aarch64.tar.zst
6570
retention-days: 5
6671

72+
# Export kernel.c + ABI_VERSION for the macOS job. kernel.c contains
73+
# the aarch64 Linux kernel as a byte array — it is OS-agnostic and can
74+
# be compiled into a .dylib by Apple's cc without rebuilding the kernel.
75+
- name: Upload kernel.c for macOS build
76+
uses: actions/upload-artifact@v4
77+
with:
78+
name: kernel-c-arm64
79+
path: |
80+
target/libkrun-build/kernel.c
81+
target/libkrun-build/ABI_VERSION
82+
retention-days: 1
83+
6784
# ---------------------------------------------------------------------------
6885
# Linux AMD64 — native kernel + libkrun build
6986
# ---------------------------------------------------------------------------
@@ -103,31 +120,31 @@ jobs:
103120
retention-days: 5
104121

105122
# ---------------------------------------------------------------------------
106-
# macOS ARM64 — kernel built via krunvm, libkrun built natively
123+
# macOS ARM64 — uses pre-built kernel.c from Linux ARM64 job
107124
# ---------------------------------------------------------------------------
108125
build-runtime-macos-arm64:
109126
name: Build Runtime (macOS ARM64)
127+
needs: [build-runtime-linux-arm64]
110128
runs-on: macos-latest-xlarge
111-
timeout-minutes: 90
129+
timeout-minutes: 30
112130
steps:
113131
- uses: actions/checkout@v4
114132

115133
- name: Install dependencies
116134
run: |
117135
set -euo pipefail
118-
# Tap slp/krun first so libkrunfw and krunvm formulae are available
119-
brew tap slp/krun
120-
brew install rust lld dtc xz libkrunfw krunvm
136+
brew install rust lld dtc xz
121137
122-
- name: Build custom libkrunfw (kernel)
123-
run: crates/openshell-vm/runtime/build-custom-libkrunfw.sh
138+
- name: Download pre-built kernel.c
139+
uses: actions/download-artifact@v4
140+
with:
141+
name: kernel-c-arm64
142+
path: target/kernel-artifact
124143

125-
- name: Build portable libkrun
126-
run: tasks/scripts/vm/build-libkrun-macos.sh
144+
- name: Build libkrunfw + libkrun from pre-built kernel
145+
run: tasks/scripts/vm/build-libkrun-macos.sh --kernel-dir target/kernel-artifact
127146

128147
- name: Package runtime tarball
129-
env:
130-
CUSTOM_PROVENANCE_DIR: target/custom-runtime
131148
run: |
132149
tasks/scripts/vm/package-vm-runtime.sh \
133150
--platform darwin-aarch64 \

architecture/custom-vm-runtime.md

Lines changed: 24 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -116,23 +116,29 @@ and makes it straightforward to correlate VM behavior with a specific runtime ar
116116
```mermaid
117117
graph LR
118118
subgraph Source["crates/openshell-vm/runtime/"]
119-
BUILD["build-custom-libkrunfw.sh\nClones libkrunfw, applies config, builds"]
120119
KCONF["kernel/openshell.kconfig\nKernel config fragment"]
121120
README["README.md\nOperator documentation"]
122121
end
123122
124-
subgraph Output["target/custom-runtime/"]
125-
LIB["libkrunfw.dylib\nCustom library"]
126-
META["provenance.json\nBuild metadata"]
127-
FRAG["openshell.kconfig\nConfig fragment used"]
128-
FULL["kernel.config\nFull kernel .config"]
123+
subgraph Linux["Linux CI (build-libkrun.sh)"]
124+
BUILD_L["Build kernel + libkrunfw.so + libkrun.so"]
125+
KERNELC["kernel.c\nKernel as C byte array"]
129126
end
130127
131-
KCONF --> BUILD
132-
BUILD --> LIB
133-
BUILD --> META
134-
BUILD --> FRAG
135-
BUILD --> FULL
128+
subgraph macOS["macOS CI (build-libkrun-macos.sh)"]
129+
BUILD_M["Compile kernel.c -> libkrunfw.dylib\nBuild libkrun.dylib"]
130+
end
131+
132+
subgraph Output["target/libkrun-build/"]
133+
LIB_SO["libkrunfw.so + libkrun.so\n(Linux)"]
134+
LIB_DY["libkrunfw.dylib + libkrun.dylib\n(macOS)"]
135+
end
136+
137+
KCONF --> BUILD_L
138+
BUILD_L --> LIB_SO
139+
BUILD_L --> KERNELC
140+
KERNELC --> BUILD_M
141+
BUILD_M --> LIB_DY
136142
```
137143

138144
## Kernel Config Fragment
@@ -222,16 +228,18 @@ supported platforms. Runs on-demand or when the kernel config / pinned versions
222228

223229
| Platform | Runner | Build Method |
224230
|----------|--------|-------------|
225-
| Linux ARM64 | `build-arm64` (self-hosted) | Native `build-libkrun.sh` |
231+
| Linux ARM64 | `build-arm64` (self-hosted) | Native `build-libkrun.sh` (also exports kernel.c) |
226232
| Linux x86_64 | `build-amd64` (self-hosted) | Native `build-libkrun.sh` |
227-
| macOS ARM64 | `macos-latest-xlarge` (GitHub-hosted) | `build-custom-libkrunfw.sh` (krunvm) + `build-libkrun-macos.sh` |
233+
| macOS ARM64 | `macos-latest-xlarge` (GitHub-hosted) | `build-libkrun-macos.sh --kernel-dir` (uses pre-built kernel.c from ARM64) |
228234

229235
Artifacts: `vm-runtime-{platform}.tar.zst` containing libkrun, libkrunfw, gvproxy, and
230236
provenance metadata.
231237

232-
The macOS kernel build requires a real macOS ARM64 runner because it uses `krunvm` to
233-
compile the Linux kernel inside a Fedora VM (Hypervisor.framework). The kernel inside
234-
libkrunfw is always Linux regardless of host platform.
238+
The aarch64 Linux kernel is compiled once on the Linux ARM64 runner. The resulting
239+
`kernel.c` (a C source file containing the kernel as a byte array) is passed to the
240+
macOS job, which compiles it into `libkrunfw.dylib` with Apple's `cc`. This eliminates
241+
the need for krunvm/Fedora VM and cuts macOS CI from ~45 min to ~5 min. The kernel
242+
inside libkrunfw is always Linux regardless of host platform.
235243

236244
### VM Binary (`release-vm-dev.yml`)
237245

crates/openshell-vm/pins.env

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
# Pinned dependency versions for openshell-vm builds.
55
#
66
# This file is sourced by build-rootfs.sh and
7-
# build-custom-libkrunfw.sh. It centralises version pins and content-addressed
7+
# build-libkrun.sh. It centralises version pins and content-addressed
88
# digests so that builds are reproducible and auditable.
99
#
1010
# Environment variables override these defaults — CI and local dev workflows

crates/openshell-vm/runtime/README.md

Lines changed: 36 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
> Status: Experimental and work in progress (WIP). VM support is under active development and may change.
44
5-
This directory contains the build infrastructure for a custom `libkrunfw` runtime
5+
This directory contains the kernel config fragment for a custom `libkrunfw` runtime
66
that enables bridge CNI and netfilter support in the OpenShell gateway VM.
77

88
## Why
@@ -21,51 +21,61 @@ that enables these networking and sandboxing features.
2121

2222
```
2323
runtime/
24-
build-custom-libkrunfw.sh # Build script for custom libkrunfw
2524
kernel/
2625
openshell.kconfig # Kernel config fragment (networking + sandboxing)
2726
```
2827

29-
## Building
28+
## Build Pipeline
3029

31-
### Prerequisites
30+
The kernel is compiled on Linux CI runners. macOS reuses the pre-built `kernel.c`
31+
artifact from the Linux ARM64 build — no krunvm or Fedora VM needed.
3232

33-
- Rust toolchain
34-
- make, git, curl
35-
- On macOS: Xcode command line tools and cross-compilation tools for aarch64
33+
```
34+
Linux ARM64: builds aarch64 kernel -> .so + exports kernel.c (parallel)
35+
Linux AMD64: builds x86_64 kernel -> .so (parallel)
36+
macOS ARM64: reuses aarch64 kernel.c -> .dylib (depends on ARM64)
37+
```
38+
39+
### Build Scripts
40+
41+
| Script | Platform | What it does |
42+
|--------|----------|-------------|
43+
| `tasks/scripts/vm/build-libkrun.sh` | Linux | Builds libkrunfw + libkrun from source, exports kernel.c |
44+
| `tasks/scripts/vm/build-libkrun-macos.sh` | macOS | Compiles pre-built kernel.c into .dylib, builds libkrun |
45+
| `tasks/scripts/vm/package-vm-runtime.sh` | Any | Packages runtime tarball (libs + gvproxy + provenance) |
3646

37-
### Quick Build
47+
### Quick Build (Linux)
3848

3949
```bash
40-
# Build custom libkrunfw (clones libkrunfw repo, applies config, builds)
41-
./crates/openshell-vm/runtime/build-custom-libkrunfw.sh
50+
# Build both libkrunfw and libkrun from source
51+
tasks/scripts/vm/build-libkrun.sh
4252

4353
# Or build the full runtime from source via mise:
4454
FROM_SOURCE=1 mise run vm:setup
4555
```
4656

47-
### Output
57+
### Quick Build (macOS)
4858

49-
Build artifacts are placed in `target/custom-runtime/`:
59+
On macOS, you need a pre-built `kernel.c` from a Linux ARM64 build:
5060

51-
```
52-
target/custom-runtime/
53-
libkrunfw.dylib # The custom library
54-
libkrunfw.<version>.dylib # Version-suffixed copy
55-
provenance.json # Build metadata (commit, hash, timestamp)
56-
openshell.kconfig # The config fragment used
57-
kernel.config # Full kernel .config (for debugging)
61+
```bash
62+
# Download pre-built runtime (recommended, ~30s):
63+
mise run vm:setup
64+
65+
# Or if you have kernel.c from a Linux build:
66+
tasks/scripts/vm/build-libkrun-macos.sh --kernel-dir target/libkrun-build
5867
```
5968

60-
### Using the Custom Runtime
69+
### Output
6170

62-
```bash
63-
# Point the bundle script at the custom build and rebuild:
64-
export OPENSHELL_VM_RUNTIME_SOURCE_DIR=target/custom-runtime
65-
mise run vm:build
71+
Build artifacts are placed in `target/libkrun-build/`:
6672

67-
# Then boot the VM as usual:
68-
mise run vm
73+
```
74+
target/libkrun-build/
75+
libkrun.so / libkrun.dylib # The VMM library
76+
libkrunfw.so* / libkrunfw.dylib # Kernel firmware library
77+
kernel.c # Linux kernel as C byte array (Linux only)
78+
ABI_VERSION # ABI version number (Linux only)
6979
```
7080

7181
## Networking

0 commit comments

Comments
 (0)