[vineethsai7]

Summary

Allow a hardened macOS VM driver to boot with a matching external runtime selected by OPENSHELL_VM_RUNTIME_DIR. The driver opens both dylibs by canonical absolute path and tells libkrun the absolute firmware path before creating a VM context. This is the runtime-loading part of the managed-Mac signing request. Developer ID and notarized boot verification remain open because no Developer ID identity was available on the test host.

Related Issue

Part of #4772. The issue remains state:triage-needed.

Changes

• Resolve libkrun.dylib and libkrunfw.5.dylib inside the configured runtime directory and load them by absolute path. Remove the temporary process-wide working-directory change.
• Patch the pinned macOS libkrun build to export krun_set_fw_path. The driver calls it before the first VM context; older external runtime archives without this entry point receive a clear error.
• Keep embedded runtime extraction and Linux behavior unchanged. Update the VM driver README, runtime documentation, and debugging skill.

Design

The pinned libkrun opens firmware with dlopen("libkrunfw.5.dylib"). Hardened dyld rejects that relative lookup even if the process enters the runtime directory: relative path not allowed in hardened program. The macOS runtime build now adds a one-time setter so libkrun opens the driver's canonical absolute firmware path. The driver verifies both files resolve inside the selected directory and preloads the exact firmware file. This avoids DYLD_* settings and process-wide directory changes.

A hardened driver needs com.apple.security.hypervisor to run the VM. With ad-hoc libraries or libraries signed by a different Team ID, it also needs com.apple.security.cs.disable-library-validation; the successful ad-hoc proof used both. Apple's library validation rule permits libraries signed with the driver's Team ID, so a matching Developer ID package is expected to need only the Hypervisor entitlement, but that exact signing configuration has not been tested. The proof did not use com.apple.security.cs.allow-dyld-environment-variables.

Operator example

Build the vm-runtime-darwin-aarch64 archive from the same revision as the driver. The archive must include the patched libkrun with krun_set_fw_path; an older published archive cannot be used with this override. Sign both libraries and the driver with the same Developer ID Application identity and the hardened runtime:

mkdir -p runtime
zstd -dc vm-runtime-darwin-aarch64.tar.zst |
  tar -xf - -C runtime ./libkrun.dylib ./libkrunfw.5.dylib

identity='Developer ID Application: Your Organization (TEAMID)'
codesign --force --options runtime --timestamp -s "$identity" runtime/libkrunfw.5.dylib
codesign --force --options runtime --timestamp -s "$identity" runtime/libkrun.dylib
codesign --force --options runtime --timestamp \
  --entitlements crates/openshell-driver-vm/entitlements.plist \
  -s "$identity" openshell-driver-vm

OPENSHELL_VM_RUNTIME_DIR=/opt/example/openshell/runtime \
  openshell-gateway --config /etc/openshell/gateway.toml
openshell sandbox create --name signed-vm --from <compatible-image>

Set OPENSHELL_VM_RUNTIME_DIR in the gateway service for a persistent deployment; it is not a gateway.toml field. Keep the directory writable only by administrators. For ad-hoc or different-Team-ID libraries, add com.apple.security.cs.disable-library-validation to the driver's entitlement file.

Compatibility

The override is opt-in. Without it, the driver uses its embedded runtime. Linux behavior is unchanged. The external macOS runtime must match the driver build and export krun_set_fw_path; older archives are rejected rather than attempting a relative firmware lookup.

Release Notes

A hardened macOS VM driver can boot with a matching, separately signed runtime selected by OPENSHELL_VM_RUNTIME_DIR. Packages with ad-hoc or different-Team-ID libraries need the library-validation exception; matching Developer ID signatures remain to be verified.

Testing

• Checks appropriate to the affected code and behavior pass.
• VM crate tests pass.
• Automated macOS VM boot test added.

Through dcb up-os-4772 with nice -n 15: cargo fmt --check -p openshell-driver-vm, cargo clippy --locked -p openshell-driver-vm --all-targets -- -D warnings, and cargo test --locked -p openshell-driver-vm --all-targets passed (297 tests). The final build-script syntax check and git diff --check passed. The zero-context libkrun patch was checked against the pinned source.

On macOS 15.8 arm64 with Hypervisor available, the final driver at a5a78cb33a9daa11eb51443a39d935cd22248988 was signed ad hoc with --options runtime, Hypervisor and disable-library-validation entitlements. The patched runtime dylibs were also signed ad hoc with --options runtime. A sandbox reached Ready, and another VM executed uname -m and printed aarch64 and BOOT-OK. Upstream main at be5e49499, built and signed with the same recipe and pointed at the same external runtime, failed with Couldn't find or load libkrunfw.5.dylib and ProcessExited. See PROOF-5141.md for the ready-to-paste manual verification section.

Follow-ups

• Verify a matching-Team-ID Developer ID signed and notarized package on a managed Mac. A real Developer ID identity was unavailable for this proof.
• Add a macOS VM boot CI lane with Hypervisor.framework access.

Checklist

• Follows Conventional Commits.
• Commits are signed off (DCO).
• Architecture docs updated.

Review follow-up

Reviewed upstream PR #5141 against NVIDIA/OpenShell main on 2026-11-02. Findings and technical reasoning are in REVIEW-5141.md. The GitHub review and review-comment APIs returned no human review findings.

• P2, process-wide cwd under Tokio: The earlier 26bb2a2 commit isolated the VM worker; this head removes the cwd change entirely.
• P2, cwd restoration after a rename: The cwd restoration path and its tests are obsolete and removed with the directory change.
• P3, inherited re-exec marker: Fixed in 26bb2a2; the outer driver removes the private marker before asynchronous driver work.
• Candidate, missing unversioned firmware dylib: Not applicable. The pinned libkrun opens libkrunfw.5.dylib dynamically; the final head passes its absolute path.
• Hardened manual proof: The prior PR head failed identically to main because dyld rejected libkrun's bare-name firmware lookup. The final head a5a78cb33 boots the hardened ad-hoc VM with the absolute-path setter while main still fails the same recipe. The disposable test account and private runtime are removed after evidence capture.

Manual verification

Result: PASS. The hardened driver at a5a78cb33a9daa11eb51443a39d935cd22248988 boots an external-runtime VM; upstream main at be5e49499 fails the same recipe.

Environment and signing

• macOS 15.8, Apple silicon (arm64), Hypervisor.framework available. Both drivers were built in release mode on this host with Rust 1.95.0. Each ran behind a private VM gateway in a disposable local account.
• Both runs used the same external libkrun.dylib and libkrunfw.5.dylib, built from the pinned runtime source with this PR's firmware-path patch. nm confirms _krun_set_fw_path is exported. The gateway set OPENSHELL_VM_RUNTIME_DIR=<runtime-dir> and unset DYLD_LIBRARY_PATH and DYLD_FALLBACK_LIBRARY_PATH.
• Both drivers and both runtime dylibs were signed ad hoc with codesign --options runtime. codesign --verify passed for the final driver and both dylibs; all four displayed CodeDirectory flags 0x10002(adhoc,runtime) with no Team ID. Both drivers had exactly com.apple.security.hypervisor and com.apple.security.cs.disable-library-validation. The latter is required here because the libraries are ad hoc signed; no dyld-environment entitlement was used.
• The host reported 0 valid identities found for code signing. A real Developer ID signature, same-Team-ID library validation, and notarization were unavailable and remain unverified.

Procedure

1. Sign the external runtime dylibs ad hoc with --options runtime, and sign each release driver ad hoc with --options runtime --entitlements <hypervisor-and-library-validation.plist>.
2. For each driver, start an isolated VM gateway with OPENSHELL_VM_RUNTIME_DIR=<runtime-dir> and no DYLD_* settings.
3. Run openshell sandbox create --name h1 --from ubuntu:24.04 -- sh -c 'uname -m; echo BOOT-OK'. For the final driver, also create hready without a trailing command and run openshell sandbox list to record a persistent phase.

Before and after

Driver	Same hardened recipe and external runtime	Result
Upstream main be5e49499	ad hoc, --options runtime, Hypervisor + library-validation exception	VM exits with status 1; sandbox enters ProcessExited error
Final PR head a5a78cb33a9daa11eb51443a39d935cd22248988	ad hoc, --options runtime, Hypervisor + library-validation exception	VM boots, guest prints aarch64 and BOOT-OK; separate sandbox reaches Ready

Main gateway and client output:

Couldn't find or load libkrunfw.5.dylib
krun_start_enter failed: -2
Error: VM exited with status 1
sandbox entered error phase while provisioning: ProcessExited: VM process exited with status 1

Final driver client output:

Created sandbox: h1
[12.2s] Waiting for VM supervisor
aarch64
BOOT-OK

NAME    PHASE
hready  Ready

A direct hardened dyld probe of the prior implementation explained the difference: dlopen("libkrunfw.5.dylib") from the runtime working directory failed with relative path not allowed in hardened program. The final driver canonicalizes both dylib paths and calls the new libkrun setter with the absolute firmware path before creating the VM context. The test account, gateways, sandboxes, and private runtime were removed after evidence capture.

The matching-Team-ID Developer ID recipe is documented from macOS library-validation behavior but was not exercised in this test.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[vineethsai7]

I have read the DCO document and I hereby sign the DCO.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-03T04:27:25.810417Z	a5a78cb	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a5a78cb33a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Restore the libkrun tree before switching pinned refs

When target/libkrun-build/libkrun is reused after changing or overriding LIBKRUN_REF (the supported pin-update flow documented in crates/openshell-driver-vm/runtime/pins.env), this application leaves src/libkrun/src/lib.rs modified in the persistent clone. The next invocation attempts git checkout "$LIBKRUN_REF" before reaching the reverse-patch check, so if the new ref changes that file Git aborts and the pin cannot be verified without an undocumented manual reset; git checkout -h confirms that only -f will “force checkout (throw away local modifications).” Reverse the known patch before checkout or apply it in a disposable worktree.

Useful? React with 👍 / 👎.