Repository navigation
Conversation
This is the network for mantsana, a macOS CI VM backend replacing Tart
(see the DESIGN.md that follows), but nothing in it is macOS-specific.
A guest's virtual NIC hands us raw Ethernet frames (on macOS over a
SOCK_DGRAM socketpair to a VZFileHandleNetworkDeviceAttachment) and they
all terminate in a gVisor netstack owned by the guest's Net, so a guest
can only reach what we connect it to, and can't see other guests:
- The gateway, IMDS (169.254.169.254) and DNS (169.254.169.253)
addresses are on our NIC, so gVisor answers ARP and ICMP for them,
and the guest needs no route or DNS changes to reach them. ARP for
any other address is dropped: the NIC is promiscuous, and answering
the guest's DHCP duplicate-address probe makes it decline its lease.
- DHCP is answered from the frame loop with one fixed lease, the same
in every slot, so a VM memory snapshot restores anywhere.
- Each TCP SYN and UDP flow is dispatched by destination to a
registered Service (a guest-facing server, or a forward to a
host-side address), to DNS (any port 53) or NTP (any port 123)
interception, or to egress dialed from the host, subject to a deny
list. Every flow and DNS query is logged.
- The host reaches the guest only by DialGuest; there is no host
route to it.
guestnettest provides a fake guest stack for tests of code built on it.
Updates tailscale/corp#39994
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I5b359a459ae52218e97cef3b74ecc14a806e0cc7
…ader These are the on-disk VM images for mantsana, a macOS CI VM backend replacing Tart (see the DESIGN.md that follows), and a way to get them for now. An image is a directory of plain files: a manifest (hardware model, machine identifier, CPUs, memory, display, MTU), a raw sparse disk, and the macOS auxiliary storage (NVRAM). A warm subdirectory holds a saved VM memory snapshot and the disk and NVRAM it goes with, for restoring in seconds instead of cold booting. Snapshots are only usable on the host OS build and at the VM size they were saved with, so WarmUsable checks, and a failed restore can mark one stale. cmd/oci-img-dl (package ocidl) downloads the macOS VM images corp CI uses today, published to OCI registries such as ghcr.io/cirruslabs/macos-tahoe-xcode, straight into such an image. Their disks are 512 MiB chunks in Apple's LZ4 stream framing, each annotated with its uncompressed SHA-256, which is verified. Chunks download in parallel, all-zero pages are left as holes, and interrupted downloads resume. It's for development and the transition; mantsana's own image distribution will be different. Updates tailscale/corp#39994 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com> Change-Id: I54ff615054bbe0ff0e1fecfa920ffa2c947bd6e7
mantsana runs corp CI's macOS VMs with Virtualization.framework (via
github.com/Code-Hex/vz/v3) instead of Tart. DESIGN.md has the design,
the decisions so far and the behavior found on hardware.
Each VM gets a private guestnet network, an IMDS server, and its disk
over NBD from an in-process guestbd server that discards the guest's
writes, so there are no per-VM disk copies. VMs restore from a warm
memory snapshot in about 4 seconds instead of cold booting in about 8.
- imds: the guest's instance metadata server (name, boot mode,
user data, time, ready and log endpoints) at 169.254.169.254.
- mantsana/guest: one VM's lifetime, independent of the hypervisor:
network, IMDS, NBD disk, boot or restore with cold boot fallback,
and warm-up, which boots, saves the memory state and persists the
disk the state goes with. A restore that fails because the user's
session is locked doesn't mark the snapshot stale. guesttest is a
fake Backend so all of this is tested on Linux.
- mantsana/vzvm: the darwin/arm64 Backend using Virtualization.framework.
- cmd/mantsana: a development tool to warm images and self-test cold
boot, warm-up and restore on a Mac, plus probes for framework
behavior and a keep-awake LaunchAgent for Macs whose MDM enforces an
idle screen lock.
The corp ciguestlet will import mantsana/guest as a second VM backend
alongside Tart, selected per host.
Updates tailscale/corp#39994
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: Ic05719ebb6ccdc12720ddcadf0e8defdee030681
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mantsana runs corp CI's macOS VMs with Virtualization.framework (via
github.com/Code-Hex/vz/v3) instead of Tart. DESIGN.md has the design,
the decisions so far and the behavior found on hardware.
Each VM gets a private guestnet network, an IMDS server, and its disk
over NBD from an in-process guestbd server that discards the guest's
writes, so there are no per-VM disk copies. VMs restore from a warm
memory snapshot in about 4 seconds instead of cold booting in about 8.
user data, time, ready and log endpoints) at 169.254.169.254.
network, IMDS, NBD disk, boot or restore with cold boot fallback,
and warm-up, which boots, saves the memory state and persists the
disk the state goes with. A restore that fails because the user's
session is locked doesn't mark the snapshot stale. guesttest is a
fake Backend so all of this is tested on Linux.
boot, warm-up and restore on a Mac, plus probes for framework
behavior and a keep-awake LaunchAgent for Macs whose MDM enforces an
idle screen lock.
The corp ciguestlet will import mantsana/guest as a second VM backend
alongside Tart, selected per host.
Updates tailscale/corp#39994