diff --git a/docs/features/compute/gpu.md b/docs/features/compute/gpu.md new file mode 100644 index 00000000..2e1fbb89 --- /dev/null +++ b/docs/features/compute/gpu.md @@ -0,0 +1,24 @@ +# GPU + +Superphenix can attach a dedicated GPU to a virtual machine with **PCI passthrough**. The guest sees the physical card as its own PCI device and loads the vendor driver, for example the NVIDIA datacenter driver. This gives native performance for AI training and inference, rendering, and CUDA workloads. + +## GPU classes + +Users pick a **GPU class**, such as `NVIDIA RTX PRO 6000 Blackwell Server Edition`. Each AZ publishes the classes its hardware provides, so the list in the console depends on the AZ you select. + +## Behavior + +- **One GPU per VM**: a VM can have at most one GPU for now. +- **Create or edit**: choose the GPU in the **Compute** step when you create the VM, or change it from **Edit**. +- **Restart required**: adding, changing or removing a GPU takes effect at the next restart. Until then, the **GPU** card on the instance details page shows _(applies after restart)_. +- **Scheduling**: the VM only starts on a node with a free GPU of the requested class. If every card of that class is in use, the VM shows `ErrorUnschedulable` until one is released. +- **No live migration**: a VM with a passthrough GPU cannot be live-migrated. It must be stopped before maintenance on its node. + +!!! note "Guest drivers" + Superphenix attaches the device but does not install drivers. Install the GPU vendor's driver inside the guest operating system, for example with cloud-init. + +## Related + +- [Instances](instance.md) +- [Create a virtual machine](../../user-guides/virtual-machines/create-a-vm.md) +- [Enabling GPU passthrough](../../operations/compute/gpu-passthrough.md) (operators) diff --git a/docs/features/compute/index.md b/docs/features/compute/index.md index 94a7cd33..f26dcaa8 100644 --- a/docs/features/compute/index.md +++ b/docs/features/compute/index.md @@ -22,6 +22,14 @@ Superphenix Compute manages virtual machine instances and full-state instance sn [:octicons-arrow-right-24: Instance snapshots](instance-snapshot.md) +- **[:material-expansion-card: GPU](gpu.md)** + + --- + + Dedicated GPUs attached to virtual machines with PCI passthrough, chosen by GPU class. + + [:octicons-arrow-right-24: GPU](gpu.md) + ## Core Capabilities @@ -30,6 +38,7 @@ Superphenix Compute manages virtual machine instances and full-state instance sn - **Multi-Interface Networking**: Attach multiple isolated L2 subnets with automatic IPAM or static IP assignment. - **Persistent Storage**: Attach bootable root volumes, secondary block disks, CD-ROM ISOs, or OCI container disks. - **Access and Consoles**: Connect via SSH, or use the browser-based web serial terminal and graphical VNC console. +- **GPU Passthrough**: Attach a dedicated GPU to a VM, chosen from the GPU classes available in the AZ. - **Firmware Options**: Support for legacy BIOS, UEFI, Secure Boot, persistent EFI variables, and vTPM 2.0. - **Cloud-Init Initialization**: Automatic user creation, SSH key injection, and custom user data execution on first boot. diff --git a/docs/operations/compute/gpu-passthrough.md b/docs/operations/compute/gpu-passthrough.md new file mode 100644 index 00000000..34abcae5 --- /dev/null +++ b/docs/operations/compute/gpu-passthrough.md @@ -0,0 +1,86 @@ +# Enabling GPU passthrough + +This page explains how to offer [GPUs](../../features/compute/gpu.md) to users of an AZ. It takes three steps: + +1. Prepare the GPU nodes so their cards can be assigned to VMs. +2. Allow KubeVirt to hand those devices to VMs. +3. Publish **GPU classes** on the AZ controller, so users can choose them in the console. + +All configuration goes under `clusters..systemConfiguration` in the values of the `superphenix-operator` release, where `` is the AZ code. See [Applying configuration changes](../applying-configuration-changes.md). + +## Prepare the GPU nodes + +Follow steps 1 to 3 of [Talos with VM-Passthrough workloads](../../installation/deployment-guide/installing-the-os/gpu-node-installation.md#2-talos-with-vm-passthrough-workloads). They enable IOMMU, bind the GPUs to `vfio-pci` and deploy the NVIDIA GPU Operator. The next section replaces the guide's _Configure Kubevirt_ step. + +## Allow the devices in KubeVirt + +List each permitted device in the KubeVirt configuration. `pciVendorSelector` is the vendor and device ID of the card. + +```yaml +clusters: + paris-1: + systemConfiguration: + apps: + kubevirt-config: + values: + config: + configuration: + permittedHostDevices: + pciHostDevices: + # Example values for the RTX PRO 6000: replace them with your card's + # PCI ID and the resource name advertised by its device plugin. + - pciVendorSelector: "10DE:2BB5" + resourceName: "nvidia.com/GB202GL_RTX_PRO_6000_BLACKWELL_SERVER_EDITION" + # true when an external device plugin (e.g. the NVIDIA GPU Operator) + # advertises the resource; false to let KubeVirt advertise it. + externalResourceProvider: true +``` + +## Publish GPU classes + +A **GPU class** is the name users see. Map each class to the `resourceName` you allowed above: + +```yaml +clusters: + paris-1: + systemConfiguration: + apps: + superphenix-controller: + values: + config: + productsConfig: + compute: + deviceMapping: + nvidia-rtx-pro-6000-bse: + deviceName: "nvidia.com/GB202GL_RTX_PRO_6000_BLACKWELL_SERVER_EDITION" + displayName: "NVIDIA RTX PRO 6000 Blackwell Server Edition" +``` + +| Field | Required | Description | +| :------------------------------ | :------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| key (`nvidia-rtx-pro-6000-bse`) | Yes | Class identifier stored on the VMs. Use **lowercase** only: the controller lowercases map keys when it loads its configuration. Do not rename a class while VMs use it. | +| `deviceName` | Yes | KubeVirt resource name. It must match a `resourceName` in `permittedHostDevices`. Classes without it are ignored and logged at startup. | +| `displayName` | No | Label shown in the console. Defaults to the class identifier. | + +The classes appear in the console as soon as the controller restarts with the new configuration. An AZ without `deviceMapping` shows no GPU option. + +## CPU model of GPU VMs + +Superphenix gives every VM that has a GPU the **`host-model`** CPU model, instead of the cluster-wide CPU model. It does this automatically when a GPU is added, and restores the cluster model when the GPU is removed. [superphenix#245](https://github.com/super-phenix/superphenix/issues/245) explains why this model was chosen. + +This matters for GPUs with a lot of memory, such as the 96 GB RTX PRO 6000, whose largest memory region is **128 GB**: + +- UEFI VMs boot with OVMF firmware, which must place that region in its 64-bit PCI memory window before the guest starts. Legacy BIOS VMs are not affected. +- OVMF needs the CPU's **1 GB pages** (`pdpe1gb`) to map a large address space. A generic cluster CPU model such as `Broadwell-noTSX-IBRS` doesn't have them, so OVMF limits itself to 40 address bits, and its window is too small for the GPU. +- Without 1 GB pages, the region is left unassigned: + - **Windows** hangs at the UEFI boot screen once the NVIDIA driver is installed. + - **Linux** boots, but the NVIDIA driver fails with `NVRM: BAR2 is 0M @ 0x0` and `modprobe: … No such device`. +- `host-model` gives the guest the closest named CPU model to the host's (for example `EPYC-Genoa` on AMD EPYC 9365 hosts), with the host's extra features. It has 1 GB pages as long as the host CPU supports them (all recent server CPUs do: check for the `pdpe1gb` flag on each GPU node), so OVMF places the region and the driver works. + +KubeVirt resolves `host-model` on the node that starts the VM, so the GPU nodes of a class can have different CPUs. It only refuses nodes whose CPU model it considers obsolete. + +## Operational notes + +- **No live migration**: VMs with a passthrough GPU cannot be live-migrated. With the platform's `LiveMigrate` eviction strategy, KubeVirt refuses to evict them, so draining a GPU node stays blocked until they are stopped. Plan GPU node maintenance with the VM owners. +- **Memory**: a VM with a passthrough device locks all of its guest memory on the host. With the default `memoryOvercommit: 200`, the node still reserves only half of it, so the scheduler can place more of these VMs than the node's memory can hold. Overcommit applies to the whole AZ and cannot be turned off for GPU VMs only, so sizing is what protects the node: each GPU VM uses one card, so give a GPU node enough memory for one VM of the largest size per card, plus the full memory of any other VMs that can run on it. +- **Capacity**: a VM requesting a GPU shows `ErrorUnschedulable` while no card of its class is free. diff --git a/docs/user-guides/virtual-machines/create-a-vm.md b/docs/user-guides/virtual-machines/create-a-vm.md index 71902ee6..b5198a35 100644 --- a/docs/user-guides/virtual-machines/create-a-vm.md +++ b/docs/user-guides/virtual-machines/create-a-vm.md @@ -26,12 +26,24 @@ The AZ is final. To run the same workload elsewhere, create a new VM in the othe ## Step 2: Compute -1. Choose a **RunStrategy**. `Manual` (the default) creates the VM stopped, and you start it yourself. `Always` boots the VM as soon as its disks are ready and restarts it if it stops. The guide uses `Always`. +1. Choose a **RunStrategy**. It decides what happens when the VM stops: + + | RunStrategy | At creation | When the VM stops | + | :--- | :--- | :--- | + | `Manual` (default) | Started once by the API (not when deployed with GitOps) | Stays stopped until you click **Start** | + | `Always` | Started | Restarted automatically, whatever the reason | + | `RerunOnFailure` | Started | Restarted if it fails (for example, its node goes down), not after a shutdown from inside the guest | + | `Once` | Started | Never restarted | + | `Halted` | Not started | Stays stopped until you click **Start** | + + In every case except `Halted`, the VM boots as soon as its disks are ready. The guide uses `Always`. 2. Keep **Instance Type** on `linux`. 3. Pick **Cores** (1 to 32) and **RAM** (1 to 64 GB). The guide uses 2 cores and 4 GB. ![Compute step with RunStrategy Always, 2 cores and 4 GB](../../assets/screenshots/virtual-machines/create-vm-compute.png) +If an administrator has published GPU classes for the AZ, a **GPU** card appears below **Compute Resources**. Keep it on **None** for this guide, or pick a GPU class to attach a dedicated GPU. See [GPU](../../features/compute/gpu.md) for how it works. + **Advanced options** lets you set CPU topology and other KubeVirt settings. Leave it collapsed for a first VM. ## Step 3: Storage & SSH @@ -92,7 +104,9 @@ You can add several subnets. Each one becomes a network interface on the VM, num The console opens the instance details page. The status starts at `Pending` while the disk imports the image. Open the **Storage** tab to follow the **Progress** column. -With RunStrategy `Always`, the VM boots on its own once the disk is ready and the status turns to `Running`. With `Manual`, click **Start** at the top of the page when the disk is ready. +The VM boots on its own once the disk is ready, and the status turns to `Running`. This is true for every RunStrategy except `Halted`: with `Halted`, click **Start** at the top of the page when the disk is ready. + +If you picked a GPU class and no card of that class is free, the VM shows `ErrorUnschedulable` until one is released. See [GPU](../../features/compute/gpu.md). ![Instance details page with status Running](../../assets/screenshots/virtual-machines/vm-details-general.png) diff --git a/zensical.toml b/zensical.toml index a41c1769..9986704a 100644 --- a/zensical.toml +++ b/zensical.toml @@ -48,6 +48,7 @@ nav = [ { "Overview" = "features/compute/index.md" }, { "Instance" = "features/compute/instance.md" }, { "Instance Snapshot" = "features/compute/instance-snapshot.md" }, + { "GPU" = "features/compute/gpu.md" }, ]}, { "Network" = [ { "Overview" = "features/network/index.md" }, @@ -115,6 +116,9 @@ nav = [ { "Manual tuning" = "operations/kernel-optimization/manual-tuning.md" }, { "Automated tuning" = "operations/kernel-optimization/automated-tuning.md" } ]}, + { "Compute" = [ + { "Enabling GPU passthrough" = "operations/compute/gpu-passthrough.md" } + ]}, { "Network" = [ { "Configure NAT gateways" = "operations/network/configure-nat-gateways.md" }, { "Setting the MTU of subnets" = "operations/network/setting-the-mtu-of-subnets.md" },