diff --git a/README.md b/README.md index 3f2d3d6..3c0fcc9 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,7 @@ steps. Not every sample requires every tool listed below. | Send and receive messages through a confidential managed service | [Service Bus Premium](service-bus-examples/README.md) | New confidential-enabled namespace, passwordless messaging, and exact message read-back checks. | | Explore attestation in a container | [Visual attestation v2 on ACI](aci-samples/visual-attestation-demo-v2/README.md) | Direct Microsoft Azure Attestation calls and a Confidential-versus-Standard comparison. | | Deploy a confidential VM or H100 confidential GPU VM | [VM samples](vm-samples/README.md) | AMD SEV-SNP, Intel TDX, and H100 paths with sample-specific attestation and prerequisites. | +| Run a confidential H100 GPU workload on Kubernetes | [AKS confidential H100 worker](aks-samples/aks-gpu-worker/README.md) | Guide-only GPU Operator automation on the latest offered AKS node image, TensorFlow GPU validation, and strict GPU attestation before and after the workload. Compute stops automatically; delete the resource group afterward. | | Keep partner datasets separate while computing shared results | [Federated multi-party analytics](multi-party-samples/advanced-app-federated/README.md) | Each partner processes its own data in its own TEE and returns aggregates. | | Explore confidential video processing | [Automotive machine vision](automotive-machine-vision/README.md) | Confidential ACI video redaction with HTTPS, CCE policy, and an attestation gate. | @@ -80,6 +81,7 @@ must not be used to send sensitive data. | Addition | Description | |---|---| +| **[Confidential H100 AKS worker](aks-samples/aks-gpu-worker/README.md)** | Deploy a single confidential H100 worker using the Azure onboarding guide's GPU Operator path and latest offered AKS image. Includes strict GPU attestation, a GPU-verified TensorFlow workload, redacted example output, and automatic compute shutdown. Revalidated in West Europe; successful execution on a newer image is not vendor-support certification. | | **[Confidential Container Apps hello world](azure-container-apps-sample/container-apps-acc/README.md#start-here)** | Deploy Microsoft's HTML quickstart using ARM and PowerShell, explicitly bound to a dedicated DC4 confidential workload profile in UAE North. | | **[Native Azure Functions on confidential Container Apps](azure-container-apps-sample/functions-on-azure-container-apps/README.md#start-here)** | Tested Node.js 24 `GET /api/hello` JSON endpoint with fresh ACR builds, immutable image deployment, managed identity, and private Blob host storage. Includes local Azurite testing, host-health checks, and portal/API access instructions. | | **[Explicit confidential-compute validation](azure-container-apps-sample/README.md#validate-confidential-compute-placement)** | Read live app/environment resources, match the selected profile to DC4, and check active/healthy revisions. Includes expected output and portal cross-checks, with a clear distinction from cryptographic attestation. | @@ -244,6 +246,7 @@ Confidential Virtual Machine (CVM) deployment scripts: ### [AKS Samples](/aks-samples/README.md) Azure Kubernetes Service with AMD SEV-SNP confidential computing: +- **[Confidential H100 GPU worker](aks-samples/aks-gpu-worker/README.md)** - Guide-only deployment on the latest offered AKS node image, strict GPU attestation before and after a TensorFlow GPU workload, and automatic compute shutdown. See the sample for quota, support boundaries, and resource deletion. - **BuildRandomAKS.ps1** - AKS cluster with Customer Managed Keys and confidential node pools - **[AKS Virtual Nodes on Confidential AKS](/aks-samples/virtual%20nodes/README.md)** - End-to-end sample that deploys AKS with virtual nodes backed by ACI, validates hello-world scheduling on the virtual node, and links to a confidential ACI follow-on manifest. diff --git a/aks-samples/README.md b/aks-samples/README.md index ca8a96e..b27b178 100644 --- a/aks-samples/README.md +++ b/aks-samples/README.md @@ -1,13 +1,39 @@ # Azure Kubernetes Service (AKS) Confidential Computing Samples -**Last Updated:** February 2026 +**Last Updated:** November 2026 ## Overview -Deploy Azure Kubernetes Service clusters with AMD SEV-SNP confidential node pools and Customer Managed Keys (CMK). +Deploy Azure Kubernetes Service clusters with confidential CPU node pools or a +confidential NVIDIA H100 GPU worker. The CPU sample includes Customer Managed +Keys (CMK); the GPU sample follows the Azure confidential H100 onboarding guide. + +## Choose a sample + +| Sample | What it covers | +|---|---| +| [Confidential H100 GPU worker](aks-gpu-worker/README.md) | Single H100 worker on the latest offered AKS node image, guide-only GPU Operator installation, TensorFlow execution verified on GPU, and strict GPU attestation before and after the workload. | +| [Confidential CPU node pools with CMK](#buildrandomaksps1) | AMD SEV-SNP confidential node pools, Key Vault, and customer-managed encryption keys. | +| [AKS virtual nodes](virtual%20nodes/README.md) | Virtual nodes backed by ACI, hello-world scheduling validation, and a confidential ACI follow-on manifest. | + +### Start with the confidential H100 worker + +See the [H100 worker guide](aks-gpu-worker/README.md) for prerequisites, +preflight and deployment commands, redacted attestation output, and cleanup. +The sample was revalidated in West Europe using an Azure-FDE kernel at least +`6.8.0-1071-azure-fde`. Successful execution on a newer AKS image is not +Microsoft/NVIDIA support certification; the guide explains that boundary. +GPU attestation is not a separate CPU/whole-VM attestation claim. + +The worker sample stops newly created compute automatically, but does **not** +delete resources. Delete its resource group when finished to remove retained +disks, IPs, and other potentially billable resources. ## Architecture +The following architecture describes the confidential CPU node-pool/CMK sample. +For the H100 configuration, use the [GPU worker guide](aks-gpu-worker/README.md). + ``` ┌─────────────────────────────────────────────────────────────────┐ │ AKS Cluster with Confidential Nodes │ diff --git a/aks-samples/aks-gpu-worker/.gitignore b/aks-samples/aks-gpu-worker/.gitignore new file mode 100644 index 0000000..a47f082 --- /dev/null +++ b/aks-samples/aks-gpu-worker/.gitignore @@ -0,0 +1 @@ +.local/ diff --git a/aks-samples/aks-gpu-worker/Common.ps1 b/aks-samples/aks-gpu-worker/Common.ps1 new file mode 100644 index 0000000..f62ac42 --- /dev/null +++ b/aks-samples/aks-gpu-worker/Common.ps1 @@ -0,0 +1,222 @@ +#Requires -Version 7.2 +. (Join-Path $PSScriptRoot 'Infrastructure.ps1') +. (Join-Path $PSScriptRoot 'Preflight.ps1') + +function New-SampleSshKey { + param([Parameter(Mandatory)][string]$Path) + $start = [System.Diagnostics.ProcessStartInfo]::new() + $start.FileName = (Get-Command ssh-keygen -CommandType Application -ErrorAction Stop).Source + $start.UseShellExecute = $false + $start.RedirectStandardError = $true + # ArgumentList preserves an empty passphrase, including on PowerShell 7.2. + foreach ($argument in @('-t', 'rsa', '-b', '3072', '-N', '', '-f', $Path, '-q')) { + $start.ArgumentList.Add($argument) + } + $process = [System.Diagnostics.Process]::Start($start) + try { + $errorText = $process.StandardError.ReadToEnd() + $process.WaitForExit() + if ($process.ExitCode -ne 0) { throw "ssh-keygen failed with exit code $($process.ExitCode): $errorText" } + if (-not (Test-Path $Path) -or -not (Test-Path "$Path.pub")) { throw 'ssh-keygen did not create the key pair.' } + } + finally { $process.Dispose() } +} + +function Assert-MinimumKernel { + param([string]$Kernel, [string]$Minimum = '6.8.0-1071-azure-fde') + $pattern = '^(\d+)\.(\d+)\.(\d+)-(\d+)-azure-fde$' + if ($Kernel -notmatch $pattern) { throw "Not an Azure FDE kernel: $Kernel" } + $actual = [version]::new([int]$Matches[1], [int]$Matches[2], [int]$Matches[3], [int]$Matches[4]) + if ($Minimum -notmatch $pattern) { throw "Invalid minimum FDE kernel: $Minimum" } + $floor = [version]::new([int]$Matches[1], [int]$Matches[2], [int]$Matches[3], [int]$Matches[4]) + if ($actual -lt $floor) { throw "Kernel $Kernel is older than $Minimum." } +} + +function Get-LatestKubernetesPatch { + param([object]$Versions) + $patches = @($Versions.values | Where-Object { $_.isPreview -ne $true } | + ForEach-Object { $_.patchVersions.PSObject.Properties.Name } | + Where-Object { $_ -match '^\d+\.\d+\.\d+$' } | + Sort-Object { [version]$_ } -Descending) + if (-not $patches.Count) { throw 'No stable regional Kubernetes patch was returned.' } + return $patches[0] +} + +function Wait-SampleAksReady { + param( + [string]$Az, [string]$Subscription, [string]$ResourceGroup, [string]$ClusterName, + [int]$TimeoutSeconds = 1200, [int]$PollSeconds = 15 + ) + $scope = @('--subscription', $Subscription, '--only-show-errors') + $deadline = [DateTime]::UtcNow.AddSeconds($TimeoutSeconds) + do { + $cluster = Invoke-CheckedCommand $Az (@('aks', 'show', '-g', $ResourceGroup, + '-n', $ClusterName, '-o', 'json') + $scope) | ConvertFrom-Json + $pools = @(Invoke-CheckedCommand $Az (@('aks', 'nodepool', 'list', '-g', $ResourceGroup, + '--cluster-name', $ClusterName, '-o', 'json') + $scope) | ConvertFrom-Json) + $states = @($cluster.provisioningState) + @($pools | ForEach-Object provisioningState) + if (@($states | Where-Object { $_ -in @('Failed', 'Canceled') }).Count) { + throw "AKS provisioning failed: $($states -join ', ')." + } + if ($pools.Count -gt 0 -and -not @($states | Where-Object { $_ -ne 'Succeeded' }).Count) { return } + if ([DateTime]::UtcNow -ge $deadline) { break } + Start-Sleep -Seconds $PollSeconds + } while ($true) + throw "AKS operations did not settle within $TimeoutSeconds seconds: $($states -join ', ')." +} + +function Stop-SampleAks { + param( + [string]$Az, [string]$Subscription, [string]$ResourceGroup, [string]$ClusterName, + [int]$TimeoutSeconds = 1200, [int]$PollSeconds = 15 + ) + $arguments = @('aks', 'stop', '-g', $ResourceGroup, '-n', $ClusterName, + '-o', 'none', '--subscription', $Subscription, '--only-show-errors') + $deadline = [DateTime]::UtcNow.AddSeconds($TimeoutSeconds) + $PSNativeCommandUseErrorActionPreference = $false + do { + $output = & $Az @arguments 2>&1 + if ($LASTEXITCODE -eq 0) { return } + $message = $output -join "`n" + # AKS can report Succeeded while an addon handler still holds its lock. + if ($message -notmatch 'OperationNotAllowed' -or $message -notmatch "in-progress .* operation") { + throw "AKS stop failed: $message" + } + if ([DateTime]::UtcNow -ge $deadline) { throw "AKS stop remained blocked for $TimeoutSeconds seconds: $message" } + Write-Warning 'AKS stop is blocked by an in-progress operation; retrying within the shutdown deadline.' + Start-Sleep -Seconds $PollSeconds + } while ($true) +} + +function Get-ExactDriverImages { + param([string]$Kernel, [string[]]$Branches) + Assert-MinimumKernel $Kernel + $token = Invoke-RestMethod 'https://nvcr.io/token?service=registry&scope=repository:nvidia/driver:pull' + foreach ($branch in ($Branches | Sort-Object { [int]$_ } -Descending -Unique)) { + if ($branch -notmatch '^\d+$') { throw "Invalid driver branch: $branch" } + $tag = "$branch-$Kernel-ubuntu24.04" + $response = Invoke-WebRequest "https://nvcr.io/v2/nvidia/driver/manifests/$tag" -Headers @{ + Authorization = "Bearer $($token.token)" + Accept = 'application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json' + } -SkipHttpErrorCheck + $available = [int]$response.StatusCode -eq 200 + $digest = $null + if ($available) { + $digest = @($response.Headers['Docker-Content-Digest'])[0] + if ($digest -notmatch '^sha256:[0-9a-f]{64}$') { throw "Missing registry digest for $tag." } + } + elseif ([int]$response.StatusCode -eq 404) { + $body = $response.Content | ConvertFrom-Json + if ($body.errors[0].code -ne 'MANIFEST_UNKNOWN' -or $body.errors[0].message -ne 'manifest unknown') { + throw "Unexpected registry 404 for $tag." + } + } + else { throw "Registry HTTP $($response.StatusCode) for $tag; not a missing-image result." } + [pscustomobject]@{ Tag = $tag; Available = $available; Digest = $digest; HttpStatus = [int]$response.StatusCode } + } +} + +function Get-GuideOperatorValues { + param([string]$Kernel) + Assert-MinimumKernel $Kernel + @" +driver: + version: 580-$Kernel + kernelModuleType: open +daemonsets: + tolerations: + - key: sku + operator: Equal + value: gpu + effect: NoSchedule +node-feature-discovery: + worker: + tolerations: + - key: sku + operator: Equal + value: gpu + effect: NoSchedule +"@ +} + +function Get-GuideWorkloadManifest { + param([Parameter(Mandatory)][string]$Manifest) + $Manifest = $Manifest.Replace("`r`n", "`n") + foreach ($required in @('name: tf-matmul-demo', 'image: tensorflow/tensorflow:2.15.0-gpu', + "spec:`n template:", ' restartPolicy: OnFailure', ' c = tf.matmul(a, b)')) { + if (-not $Manifest.Contains($required)) { throw "Upstream workload manifest changed: missing $required." } + } + $Manifest = $Manifest.Replace("spec:`n template:", "spec:`n backoffLimit: 0`n activeDeadlineSeconds: 900`n template:") + $Manifest = $Manifest.Replace(' restartPolicy: OnFailure', @' + restartPolicy: Never + runtimeClassName: nvidia + nodeSelector: + agentpool: gpunp +'@) + $Manifest = $Manifest.Replace(' c = tf.matmul(a, b)', @' + tf.config.set_soft_device_placement(False) + assert tf.config.list_physical_devices('GPU'), "No GPU available" + with tf.device('/GPU:0'): + c = tf.matmul(a, b) + assert 'GPU:0' in c.device, "Matmul did not execute on GPU" + print("GPU matmul device:", c.device) +'@) + return $Manifest +} + +function Stop-ExistingNccCompute { + param([string]$Az, [string]$Subscription, [string]$Location, [switch]$Stop) + $scope = @('--subscription', $Subscription, '--only-show-errors') + $clusters = @(Invoke-CheckedCommand $Az (@('aks', 'list', '-o', 'json') + $scope) | ConvertFrom-Json) + $sets = @(Invoke-CheckedCommand $Az (@('vmss', 'list', '-o', 'json') + $scope) | ConvertFrom-Json) + $vms = @(Invoke-CheckedCommand $Az (@('vm', 'list', '--show-details', '-o', 'json') + $scope) | ConvertFrom-Json) + foreach ($set in @($sets | Where-Object { $_.location -eq $Location -and $_.sku.name -like 'Standard_NCC*' })) { + $owner = @($clusters | Where-Object { $_.nodeResourceGroup -eq $set.resourceGroup }) + if ($owner.Count -gt 1) { throw "Ambiguous AKS owner for $($set.name)." } + if ($owner.Count -eq 1) { + $cluster = $owner[0] + Write-Host "Existing NCC: $($cluster.name), power $($cluster.powerState.code), capacity $($set.sku.capacity)." + if ($cluster.powerState.code -ne 'Stopped') { + if (-not $Stop) { throw 'Running NCC AKS found. Pass -StopExistingNcc to authorize shutdown.' } + Stop-SampleAks $Az $Subscription $cluster.resourceGroup $cluster.name + $state = Invoke-CheckedCommand $Az (@('aks', 'show', '-g', $cluster.resourceGroup, '-n', $cluster.name, + '--query', 'powerState.code', '-o', 'tsv') + $scope) + if ($state -ne 'Stopped') { throw "AKS $($cluster.name) did not stop." } + } + $instances = @(Invoke-CheckedCommand $Az (@('vmss', 'list-instances', '-g', $set.resourceGroup, + '-n', $set.name, '--expand', 'instanceView', '-o', 'json') + $scope) | ConvertFrom-Json) + foreach ($instance in $instances) { + $power = @($instance.instanceView.statuses | Where-Object code -Like 'PowerState/*') + if ($power.Count -ne 1 -or $power[0].code -ne 'PowerState/deallocated') { + throw "Existing AKS NCC instance $($instance.name) is not deallocated." + } + } + } + else { + $instances = @(Invoke-CheckedCommand $Az (@('vmss', 'list-instances', '-g', $set.resourceGroup, + '-n', $set.name, '--expand', 'instanceView', '-o', 'json') + $scope) | ConvertFrom-Json) + $active = @($instances | Where-Object { + 'PowerState/deallocated' -notin @($_.instanceView.statuses.code) + }) + if ($active.Count) { + if (-not $Stop) { throw "Running standalone NCC scale set $($set.name). Pass -StopExistingNcc." } + Invoke-CheckedCommand $Az (@('vmss', 'deallocate', '-g', $set.resourceGroup, '-n', $set.name, '-o', 'none') + $scope) + $verified = @(Invoke-CheckedCommand $Az (@('vmss', 'list-instances', '-g', $set.resourceGroup, + '-n', $set.name, '--expand', 'instanceView', '-o', 'json') + $scope) | ConvertFrom-Json) + if (@($verified | Where-Object { 'PowerState/deallocated' -notin @($_.instanceView.statuses.code) }).Count) { + throw "Standalone NCC scale set $($set.name) did not deallocate." + } + } + } + } + foreach ($vm in @($vms | Where-Object { $_.location -eq $Location -and $_.hardwareProfile.vmSize -like 'Standard_NCC*' })) { + Write-Host "Existing NCC VM: $($vm.name), $($vm.powerState)." + if ($vm.powerState -ne 'VM deallocated') { + if (-not $Stop) { throw "Existing NCC VM $($vm.name) needs -StopExistingNcc." } + Invoke-CheckedCommand $Az (@('vm', 'deallocate', '--ids', $vm.id, '-o', 'none') + $scope) + $state = Invoke-CheckedCommand $Az (@('vm', 'get-instance-view', '--ids', $vm.id, + '--query', "instanceView.statuses[?starts_with(code, 'PowerState/')].code | [0]", '-o', 'tsv') + $scope) + if ($state -ne 'PowerState/deallocated') { throw "VM $($vm.name) did not deallocate." } + } + } +} diff --git a/aks-samples/aks-gpu-worker/Deploy.ps1 b/aks-samples/aks-gpu-worker/Deploy.ps1 new file mode 100644 index 0000000..e797119 --- /dev/null +++ b/aks-samples/aks-gpu-worker/Deploy.ps1 @@ -0,0 +1,291 @@ +#Requires -Version 7.2 +[CmdletBinding()] +param( + [Parameter(Mandatory)][guid]$SubscriptionId, + [ValidatePattern('^[a-zA-Z0-9_-]+$')][string]$ResourceGroup = "rg-aks-gpu-worker-$(Get-Date -Format yyyyMMddHHmmss)", + [ValidatePattern('^[a-zA-Z0-9-]+$')][string]$ClusterName = "aks-gpu-worker-$(Get-Date -Format yyyyMMddHHmmss)", + [string]$Location = 'westeurope', + [string]$KubernetesVersion, + [string]$MinimumKernel = '6.8.0-1071-azure-fde', + [string]$OutputDirectory = (Join-Path $PSScriptRoot '.local'), + [switch]$StopExistingNcc, + [switch]$PreflightOnly, + [switch]$KeepRunning +) +. (Join-Path $PSScriptRoot 'Common.ps1') +$oldExtensions = $env:AZURE_EXTENSION_DIR +$oldDynamicInstall = $env:AZURE_EXTENSION_USE_DYNAMIC_INSTALL +$created = $false +$result = [ordered]@{ + StartedAtUtc = [DateTime]::UtcNow.ToString('o') + SubscriptionId = $SubscriptionId.ToString() + ResourceGroup = $ResourceGroup + ClusterName = $ClusterName + Location = $Location + Status = 'Preparing' + VendorSupportValidated = $false + DriverPath = 'Guide GPU Operator v25.3.1, NVIDIA branch 580 only' +} + +function Invoke-Az { + param([string[]]$Arguments) + Invoke-CheckedCommand $az ($Arguments + $scope) +} + +function Invoke-Kube { + param([string[]]$Arguments) + Invoke-CheckedCommand kubectl (@('--kubeconfig', $kubeConfig) + $Arguments) +} + +function Wait-GpuJob { + param([string]$Name, [string]$Namespace, [int]$TimeoutSeconds = 600) + $deadline = [DateTime]::UtcNow.AddSeconds($TimeoutSeconds) + do { + $job = Invoke-Kube @('get', 'job', $Name, '-n', $Namespace, '-o', 'json') | ConvertFrom-Json + $property = $job.status.PSObject.Properties['conditions'] + $conditions = @(if ($property) { $property.Value }) + if (@($conditions | Where-Object { $_ -and $_.type -eq 'Failed' -and $_.status -eq 'True' }).Count) { + Save-GpuJobDiagnostics $Name $Namespace + throw "Job $Namespace/$Name failed. Inspect its pods using the dedicated kubeconfig." + } + if (@($conditions | Where-Object { $_ -and $_.type -eq 'Complete' -and $_.status -eq 'True' }).Count) { return } + Start-Sleep 10 + } while ([DateTime]::UtcNow -lt $deadline) + Save-GpuJobDiagnostics $Name $Namespace + throw "Job $Namespace/$Name did not complete within $TimeoutSeconds seconds." +} + +function Save-GpuJobDiagnostics { + param([string]$Name, [string]$Namespace) + try { + $pods = Invoke-Kube @('get', 'pods', '-n', $Namespace, '-l', "job-name=$Name", '-o', 'json') + $pods | Set-Content (Join-Path $runDirectory "$Name-failed-pods.json") + $log = Invoke-Kube @('logs', '-n', $Namespace, "job/$Name", '--tail=200') + (($log -join "`n") -replace '(?s)Entity Attestation Token:.*', '[Attestation token omitted]') | + Set-Content (Join-Path $runDirectory "$Name-failed-log.txt") + } + catch { Write-Warning "Unable to capture complete diagnostics for ${Namespace}/${Name}: $($_.Exception.Message)" } +} + +try { + Assert-MinimumKernel $MinimumKernel + $az = Resolve-AzureCli + foreach ($tool in @('kubectl', 'helm', 'ssh-keygen')) { Get-Command $tool -CommandType Application -ErrorAction Stop | Out-Null } + New-Item -ItemType Directory -Path $OutputDirectory -Force | Out-Null + $OutputDirectory = (Resolve-Path $OutputDirectory).Path + $runDirectory = Join-Path $OutputDirectory "$ClusterName-$(Get-Date -Format yyyyMMddHHmmssfff)" + New-Item -ItemType Directory -Path $runDirectory | Out-Null + $kubeConfig = Join-Path $runDirectory 'kubeconfig' + $scope = @('--subscription', $SubscriptionId.ToString(), '--only-show-errors') + $env:AZURE_EXTENSION_DIR = Join-Path $runDirectory 'cli-extensions-core' + $env:AZURE_EXTENSION_USE_DYNAMIC_INSTALL = 'no' + New-Item -ItemType Directory -Path $env:AZURE_EXTENSION_DIR | Out-Null + $result.EvidenceDirectory = $runDirectory + + $versions = Invoke-Az @('aks', 'get-versions', '-l', $Location, '-o', 'json') | ConvertFrom-Json + if (-not $KubernetesVersion) { + $default = @($versions.values | Where-Object { $_.isDefault -eq $true -and $_.isPreview -ne $true }) + if ($default.Count -ne 1) { throw 'Expected exactly one non-preview default AKS minor version.' } + $KubernetesVersion = Get-LatestKubernetesPatch ([pscustomobject]@{ values = $default }) + } + $stable = @($versions.values | Where-Object { $_.isPreview -ne $true } | + ForEach-Object { $_.patchVersions.PSObject.Properties.Name }) + if ($KubernetesVersion -notin $stable) { throw "Non-preview Kubernetes $KubernetesVersion is not offered." } + $result.KubernetesVersion = $KubernetesVersion + $exists = Invoke-Az @('group', 'exists', '-n', $ResourceGroup, '-o', 'tsv') + if ($exists -ne 'false') { throw 'Use a new resource group; this script never adopts or overwrites existing resources.' } + + if (-not $PreflightOnly) { + Stop-ExistingNccCompute -Az $az -Subscription $SubscriptionId -Location $Location -Stop:$StopExistingNcc + } + Test-SamplePreflight -Az $az -Subscription $SubscriptionId -Location $Location ` + -KubernetesVersion $KubernetesVersion -OutputDirectory (Join-Path $runDirectory 'preflight') + Invoke-CheckedCommand helm @('pull', 'gpu-operator', '--repo', 'https://helm.ngc.nvidia.com/nvidia', + '--version', 'v25.3.1', '--destination', $runDirectory) + $chartPath = Join-Path $runDirectory 'gpu-operator-v25.3.1.tgz' + $result.ChartSha256 = (Get-FileHash $chartPath -Algorithm SHA256).Hash + if ($PreflightOnly) { + $result.Status = 'PreflightPassed' + return + } + + $sshKey = Join-Path $runDirectory 'ssh-key' + New-SampleSshKey -Path $sshKey + Invoke-Az @('group', 'create', '-n', $ResourceGroup, '-l', $Location, '--tags', 'sample=aks-gpu-worker', '-o', 'none') + $created = $true + Invoke-Az @('aks', 'create', '-g', $ResourceGroup, '-n', $ClusterName, '-l', $Location, + '--kubernetes-version', $KubernetesVersion, '--nodepool-name', 'sysnp', '--node-count', '1', + '--node-vm-size', 'Standard_D4ds_v6', '--os-sku', 'Ubuntu2404', '--ssh-key-value', "$sshKey.pub", + '--enable-managed-identity', '--network-plugin', 'azure', '--network-plugin-mode', 'overlay', + '--auto-upgrade-channel', 'none', '--node-os-upgrade-channel', 'None', + '--tags', 'sample=aks-gpu-worker', '-o', 'none') + Wait-SampleAksReady $az $SubscriptionId $ResourceGroup $ClusterName + Invoke-Az @('aks', 'nodepool', 'add', '-g', $ResourceGroup, '--cluster-name', $ClusterName, + '-n', 'gpunp', '--node-count', '1', '--node-vm-size', 'Standard_NCC40ads_H100_v5', + '--os-sku', 'Ubuntu2404', '--gpu-driver', 'None', '--node-taints', 'sku=gpu:NoSchedule', '-o', 'none') + Wait-SampleAksReady $az $SubscriptionId $ResourceGroup $ClusterName + + foreach ($poolName in @('sysnp', 'gpunp')) { + $pool = Invoke-Az @('aks', 'nodepool', 'show', '-g', $ResourceGroup, '--cluster-name', $ClusterName, + '-n', $poolName, '-o', 'json') | ConvertFrom-Json + $upgrades = Invoke-Az @('aks', 'nodepool', 'get-upgrades', '-g', $ResourceGroup, + '--cluster-name', $ClusterName, '-n', $poolName, '-o', 'json') | ConvertFrom-Json + if (-not $upgrades.latestNodeImageVersion) { throw "AKS did not report the latest image for $poolName." } + if ($pool.nodeImageVersion -ne $upgrades.latestNodeImageVersion) { + Invoke-Az @('aks', 'nodepool', 'upgrade', '-g', $ResourceGroup, '--cluster-name', $ClusterName, + '-n', $poolName, '--node-image-only', '--yes', '-o', 'none') + $pool = Invoke-Az @('aks', 'nodepool', 'show', '-g', $ResourceGroup, '--cluster-name', $ClusterName, + '-n', $poolName, '-o', 'json') | ConvertFrom-Json + } + if ($pool.nodeImageVersion -ne $upgrades.latestNodeImageVersion) { throw "$poolName is not on the latest offered image." } + $result["${poolName}Image"] = $pool.nodeImageVersion + } + $pools = @(Invoke-Az @('aks', 'nodepool', 'list', '-g', $ResourceGroup, + '--cluster-name', $ClusterName, '-o', 'json') | ConvertFrom-Json) + Assert-SamplePools -Pools $pools -SystemVmSize Standard_D4ds_v6 + $cluster = Invoke-Az @('aks', 'show', '-g', $ResourceGroup, '-n', $ClusterName, '-o', 'json') | ConvertFrom-Json + $sets = @(Invoke-Az @('vmss', 'list', '-g', $cluster.nodeResourceGroup, '-o', 'json') | ConvertFrom-Json) + Assert-ConfidentialGpuScaleSet $sets + $sets | ConvertTo-Json -Depth 40 | Set-Content (Join-Path $runDirectory 'scale-sets.json') + Invoke-Az @('aks', 'get-credentials', '-g', $ResourceGroup, '-n', $ClusterName, '--file', $kubeConfig, '-o', 'none') + Invoke-Kube @('wait', 'node', '-l', 'agentpool=gpunp', '--for=condition=Ready', '--timeout=900s') + $node = Get-GpuNode -KubeConfig $kubeConfig + $result.Kernel = $node.status.nodeInfo.kernelVersion + $result.OsImage = $node.status.nodeInfo.osImage + if ($result.OsImage -notlike 'Ubuntu 24.04*') { throw "Unexpected GPU OS: $($result.OsImage)." } + Assert-MinimumKernel $result.Kernel $MinimumKernel + $node | ConvertTo-Json -Depth 30 | Set-Content (Join-Path $runDirectory 'gpu-node.json') + + $images = @(Get-ExactDriverImages -Kernel $result.Kernel -Branches @('580')) + $images | ConvertTo-Json -Depth 5 | Set-Content (Join-Path $runDirectory 'driver-images.json') + $image = @($images | Where-Object Available) + if ($image.Count -ne 1) { + $result.Status = 'BlockedMissingGuideDriver' + throw "The guide's image nvcr.io/nvidia/driver:580-$($result.Kernel)-ubuntu24.04 is absent (MANIFEST_UNKNOWN). No alternative driver or kernel will be installed." + } + $result.DriverImage = "nvcr.io/nvidia/driver:$($image[0].Tag)" + $result.DriverDigest = $image[0].Digest + $values = Get-GuideOperatorValues $result.Kernel + $valuesPath = Join-Path $runDirectory 'operator-values.yaml' + $values | Set-Content $valuesPath + Invoke-CheckedCommand helm @('install', 'gpu-operator', $chartPath, '--kubeconfig', $kubeConfig, + '--namespace', 'gpu-operator', '--create-namespace', '-f', $valuesPath, '--wait', '--timeout', '20m') + $deadline = [DateTime]::UtcNow.AddMinutes(10) + do { + $node = Get-GpuNode -KubeConfig $kubeConfig + $gpu = $node.status.allocatable.PSObject.Properties['nvidia.com/gpu'] + if ($gpu -and $gpu.Value -eq '1') { break } + Start-Sleep 10 + } while ([DateTime]::UtcNow -lt $deadline) + if (-not $gpu -or $gpu.Value -ne '1') { throw 'The worker did not expose exactly one allocatable GPU.' } + $driverPods = (Invoke-Kube @('get', 'pods', '-n', 'gpu-operator', '-l', 'app=nvidia-driver-daemonset', '-o', 'json') | + ConvertFrom-Json).items + $driver = @($driverPods | Where-Object { $_.spec.nodeName -eq $node.metadata.name }) + if ($driver.Count -ne 1) { throw 'Expected one GPU driver pod on the confidential worker.' } + $driverName = $driver[0].metadata.name + foreach ($operation in @('-f', '-e', '-srs', '-q')) { + $command = @('exec', '-n', 'gpu-operator', $driverName, '-c', 'nvidia-driver-ctr', '--', 'nvidia-smi', 'conf-compute', $operation) + if ($operation -eq '-srs') { $command += '1' } + $output = Invoke-Kube $command + $output | Set-Content (Join-Path $runDirectory "cc-$($operation.TrimStart('-')).txt") + if ($operation -eq '-f' -and "$output" -notmatch 'CC status: ON') { throw 'GPU confidential mode is not ON.' } + if ($operation -eq '-e' -and "$output" -notmatch 'CC Environment: PRODUCTION') { throw 'GPU is not in PRODUCTION mode.' } + if ($operation -eq '-q' -and "$output" -notmatch 'CC GPUs Ready State\s*: Ready') { throw 'GPU is not Ready.' } + } + + $guideCommit = '136262ef00ad75b8e0bc789563f043006fb68f4c' + $manifestUrl = "https://raw.githubusercontent.com/Azure/az-cgpu-onboarding/$guideCommit/docs/cgpu-aks-attestation.yaml" + $attestation = (Invoke-WebRequest $manifestUrl).Content + if ($attestation -notmatch 'python3 -m verifier.cc_admin') { throw 'Upstream attestation manifest changed; review it before continuing.' } + $identity = Invoke-Kube @('exec', '-n', 'gpu-operator', $driverName, '-c', 'nvidia-driver-ctr', + '--', 'nvidia-smi', '--query-gpu=uuid', '--format=csv,noheader') + $uuid = ("$identity").Trim() + if ($uuid -notmatch '^GPU-[0-9a-fA-F-]+$') { throw 'Expected exactly one GPU UUID.' } + $result.GpuUuid = $uuid + $attestation = $attestation.Replace('./prodtest/bin/python3 -m verifier.cc_admin', + './prodtest/bin/python3 -m verifier.cc_admin --ocsp_attestation_settings strict') + # Pin scheduling and deadline while preserving the guide's verifier installation. + $attestation = $attestation.Replace(' backoffLimit: 0', " backoffLimit: 0`n activeDeadlineSeconds: 900") + $attestation = $attestation.Replace(' restartPolicy: Never', @" + restartPolicy: Never + nodeSelector: + agentpool: gpunp + tolerations: + - key: sku + operator: Equal + value: gpu + effect: NoSchedule +"@) + $attestationPath = Join-Path $runDirectory 'attestation.yaml' + $attestation | Set-Content $attestationPath + foreach ($phase in @('before', 'after')) { + Invoke-Kube @('apply', '-f', $attestationPath) + Wait-GpuJob -Name local-gpu-attestation -Namespace attest -TimeoutSeconds 950 + $log = Invoke-Kube @('logs', '-n', 'attest', 'job/local-gpu-attestation') + $text = $log -join "`n" + $redacted = $text -replace '(?s)Entity Attestation Token:.*', '[Attestation token omitted]' + $redacted | Set-Content (Join-Path $runDirectory "attestation-$phase.txt") + if ($text -notmatch 'GPU Attestation is Successful\.') { throw "GPU attestation $phase did not report success." } + if ($text -notmatch "(?i)$([regex]::Escape($uuid))") { throw "Attestation $phase did not identify this worker's GPU." } + $result["Attestation${phase}Passed"] = $true + Invoke-Kube @('delete', 'job', 'local-gpu-attestation', '-n', 'attest', '--wait=true') + if ($phase -eq 'before') { + $workloadUrl = "https://raw.githubusercontent.com/Azure/az-cgpu-onboarding/$guideCommit/docs/samples-tf-matmul-demo.yaml" + $smoke = Get-GuideWorkloadManifest (Invoke-WebRequest $workloadUrl).Content + $smokePath = Join-Path $runDirectory 'gpu-smoke-test.yaml' + $smoke | Set-Content $smokePath + Invoke-Kube @('apply', '-f', $smokePath) + Wait-GpuJob -Name tf-matmul-demo -Namespace default -TimeoutSeconds 950 + $log = Invoke-Kube @('logs', 'job/tf-matmul-demo') + $log | Set-Content (Join-Path $runDirectory 'gpu-smoke-test.txt') + if (($log -join "`n") -notmatch 'GPU matmul device:.*GPU:0') { throw 'Guide TensorFlow workload did not report GPU execution.' } + $result.GpuWorkloadPassed = $true + } + } + $result.Status = 'Succeeded' +} +catch { + $result.Error = $_.Exception.Message + if ($result.Status -notlike 'Blocked*') { $result.Status = 'Failed' } + throw +} +finally { + try { + if ($created -and -not $KeepRunning) { + $clusters = @(Invoke-Az @('aks', 'list', '-g', $ResourceGroup, '-o', 'json') | ConvertFrom-Json) + if ($clusters.Count -eq 1) { + Wait-SampleAksReady $az $SubscriptionId $ResourceGroup $ClusterName + Stop-SampleAks $az $SubscriptionId $ResourceGroup $ClusterName + $state = Invoke-Az @('aks', 'show', '-g', $ResourceGroup, '-n', $ClusterName, + '--query', 'powerState.code', '-o', 'tsv') + $result.FinalPowerState = "$state" + if ($state -ne 'Stopped') { throw 'Disposable AKS did not stop; inspect it immediately for ongoing NCC charges.' } + $nodeGroup = $clusters[0].nodeResourceGroup + $sets = @(Invoke-Az @('vmss', 'list', '-g', $nodeGroup, '-o', 'json') | ConvertFrom-Json) + foreach ($set in $sets) { + $instances = @(Invoke-Az @('vmss', 'list-instances', '-g', $nodeGroup, '-n', $set.name, + '--expand', 'instanceView', '-o', 'json') | ConvertFrom-Json) + if (@($instances | Where-Object { 'PowerState/deallocated' -notin @($_.instanceView.statuses.code) }).Count) { + throw "Scale set $($set.name) still has allocated instances after AKS stop." + } + } + $result.ComputeDeallocated = $true + } + } + elseif ($created) { $result.FinalPowerState = 'KeptRunningByRequest' } + } + catch { + $result.CleanupError = $_.Exception.Message + $result.Status = 'CleanupFailed' + throw + } + finally { + $result.FinishedAtUtc = [DateTime]::UtcNow.ToString('o') + if ($result.Contains('EvidenceDirectory')) { + $result | ConvertTo-Json -Depth 10 | Set-Content (Join-Path $runDirectory 'result.json') + Write-Host "Evidence: $runDirectory" + } + $env:AZURE_EXTENSION_DIR = $oldExtensions + $env:AZURE_EXTENSION_USE_DYNAMIC_INSTALL = $oldDynamicInstall + } +} diff --git a/aks-samples/aks-gpu-worker/Infrastructure.ps1 b/aks-samples/aks-gpu-worker/Infrastructure.ps1 new file mode 100644 index 0000000..4214009 --- /dev/null +++ b/aks-samples/aks-gpu-worker/Infrastructure.ps1 @@ -0,0 +1,80 @@ +#Requires -Version 7.2 +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +function Resolve-AzureCli { + $commands = @(Get-Command az -CommandType Application -ErrorAction SilentlyContinue) + if ($commands.Count) { return $commands[0].Source } + if ($IsWindows) { + foreach ($root in @($env:ProgramFiles, ${env:ProgramFiles(x86)})) { + if ($root) { + $candidate = Join-Path $root 'Microsoft SDKs\Azure\CLI2\wbin\az.cmd' + if (Test-Path $candidate) { return $candidate } + } + } + } + throw 'Azure CLI is required. Install it and run az login before retrying.' +} + +function Invoke-CheckedCommand { + param( + [Parameter(Mandatory)][string]$Command, + [Parameter(Mandatory)][string[]]$ArgumentList + ) + $output = & $Command @ArgumentList + if ($LASTEXITCODE -ne 0) { + throw "$Command $($ArgumentList -join ' ') failed with exit code $LASTEXITCODE." + } + return $output +} + +function Get-GpuNode { + param( + [Parameter(Mandatory)][string]$KubeConfig, + [string]$PoolName = 'gpunp' + ) + $nodes = Invoke-CheckedCommand kubectl @('--kubeconfig', $KubeConfig, 'get', 'nodes', + '-l', "agentpool=$PoolName", '-o', 'json') | ConvertFrom-Json + if (@($nodes.items).Count -ne 1) { + throw "Expected exactly one node in pool $PoolName; found $(@($nodes.items).Count)." + } + return $nodes.items[0] +} + +function Assert-SamplePools { + param( + [Parameter(Mandatory)][object[]]$Pools, + [Parameter(Mandatory)][string]$SystemVmSize, + [switch]$AllowMissingGpu + ) + $system = @($Pools | Where-Object name -EQ 'sysnp') + $gpu = @($Pools | Where-Object name -EQ 'gpunp') + if ($system.Count -ne 1 -or @($Pools | Where-Object { $_.name -notin @('sysnp', 'gpunp') }).Count -or + $gpu.Count -gt 1 -or (-not $AllowMissingGpu -and $gpu.Count -ne 1)) { + throw 'Expected one system pool and one GPU pool, with no additional pools.' + } + + foreach ($pool in $Pools) { + $size = if ($pool.name -eq 'sysnp') { $SystemVmSize } else { 'Standard_NCC40ads_H100_v5' } + $mode = if ($pool.name -eq 'sysnp') { 'System' } else { 'User' } + if ($pool.vmSize -ne $size -or $pool.count -ne 1 -or $pool.osSku -ne 'Ubuntu2404' -or + $pool.mode -ne $mode -or $pool.enableAutoScaling -eq $true -or $pool.provisioningState -ne 'Succeeded') { + throw "Pool $($pool.name) does not match the single-node sample configuration; it was not modified." + } + if ($pool.name -eq 'gpunp' -and + ($pool.gpuProfile.driver -ne 'None' -or 'sku=gpu:NoSchedule' -notin $pool.nodeTaints)) { + throw 'GPU pool must disable automatic drivers and retain the GPU scheduling taint.' + } + } +} + +function Assert-ConfidentialGpuScaleSet { + param([Parameter(Mandatory)][object[]]$ScaleSets) + $gpu = @($ScaleSets | Where-Object { $_.sku.name -eq 'Standard_NCC40ads_H100_v5' }) + if ($gpu.Count -ne 1) { throw 'Expected exactly one H100 scale set.' } + $security = $gpu[0].virtualMachineProfile.securityProfile + if ($gpu[0].sku.capacity -ne 1 -or $security.securityType -ne 'ConfidentialVM' -or + $security.uefiSettings.secureBootEnabled -ne $true -or $security.uefiSettings.vTpmEnabled -ne $true) { + throw 'H100 scale set must have one ConfidentialVM instance with Secure Boot and vTPM enabled.' + } +} diff --git a/aks-samples/aks-gpu-worker/Preflight.ps1 b/aks-samples/aks-gpu-worker/Preflight.ps1 new file mode 100644 index 0000000..c463e4e --- /dev/null +++ b/aks-samples/aks-gpu-worker/Preflight.ps1 @@ -0,0 +1,62 @@ +#Requires -Version 7.2 +function Test-SamplePreflight { + param( + [string]$Az, + [string]$Subscription, + [string]$Location, + [string]$KubernetesVersion, + [string]$OutputDirectory + ) + $scope = @('--subscription', $Subscription, '--only-show-errors') + $account = Invoke-CheckedCommand $Az (@('account', 'show', '-o', 'json') + $scope) | ConvertFrom-Json + if ($account.state -ne 'Enabled') { throw 'The target subscription is not enabled.' } + $cliVersion = (Invoke-CheckedCommand $Az @('version', '-o', 'json') | ConvertFrom-Json).'azure-cli' + Write-Host "Using built-in Azure CLI $cliVersion AKS commands in $Location (extensions isolated)." + $versions = Invoke-CheckedCommand $Az (@('aks', 'get-versions', '--location', $Location, '-o', 'json') + $scope) | ConvertFrom-Json + $patches = @($versions.values | ForEach-Object { $_.patchVersions.PSObject.Properties.Name }) + if ($KubernetesVersion -notin $patches) { throw "Kubernetes $KubernetesVersion is not offered in $Location." } + + $skus = @(Invoke-CheckedCommand $Az (@('vm', 'list-skus', '--location', $Location, + '--resource-type', 'virtualMachines', '--all', '-o', 'json') + $scope) | ConvertFrom-Json) + $usage = @(Invoke-CheckedCommand $Az (@('vm', 'list-usage', '--location', $Location, '-o', 'json') + $scope) | ConvertFrom-Json) + $required = @{} + $skuEvidence = @(foreach ($size in @('Standard_D4ds_v6', 'Standard_NCC40ads_H100_v5')) { + $sku = @($skus | Where-Object name -EQ $size) + if ($sku.Count -ne 1) { throw "SKU $size was not found in $Location." } + $sku = $sku[0] + $regionBlocks = @($sku.restrictions | Where-Object type -EQ 'Location') + if ($regionBlocks.Count) { + throw "SKU $size has a regional subscription restriction: $($regionBlocks | ConvertTo-Json -Depth 10 -Compress)" + } + $cores = [int]($sku.capabilities | Where-Object name -EQ 'vCPUs').value + if (-not $required.ContainsKey($sku.family)) { $required[$sku.family] = 0 } + $required[$sku.family] += $cores + $sku + }) + $total = 0 + foreach ($family in $required.Keys) { + $quota = @($usage | Where-Object { $_.name.value -eq $family }) + if ($quota.Count -ne 1 -or ([int]$quota[0].limit - [int]$quota[0].currentValue) -lt $required[$family]) { + throw "Insufficient $family quota for $($required[$family]) new vCPUs." + } + $total += $required[$family] + } + $regional = $usage | Where-Object { $_.name.value -eq 'cores' } + if (([int]$regional.limit - [int]$regional.currentValue) -lt $total) { + throw "Insufficient regional vCPU quota for $total cores." + } + $feature = Invoke-CheckedCommand $Az (@('feature', 'show', '--namespace', 'Microsoft.ContainerService', + '--name', 'Ubuntu2404Preview', '--query', 'properties.state', '-o', 'tsv') + $scope) + $preflight = [ordered]@{ + CheckedAtUtc = [DateTime]::UtcNow.ToString('o') + Location = $Location + KubernetesVersion = $KubernetesVersion + PreviewState = "$feature" + Skus = $skuEvidence + Quota = @($usage | Where-Object { $_.name.value -in @($required.Keys) + @('cores') }) + } + New-Item -ItemType Directory -Path $OutputDirectory -Force | Out-Null + $preflight | ConvertTo-Json -Depth 20 | Set-Content (Join-Path $OutputDirectory 'preflight.json') + if ($feature -ne 'Registered') { throw "Ubuntu2404Preview is $feature. Arrange subscription registration before deploying." } + Write-Host 'Preflight passed; no Azure resources were changed.' +} diff --git a/aks-samples/aks-gpu-worker/README.md b/aks-samples/aks-gpu-worker/README.md new file mode 100644 index 0000000..7fdea86 --- /dev/null +++ b/aks-samples/aks-gpu-worker/README.md @@ -0,0 +1,195 @@ +# Latest-image confidential H100 AKS worker + +Automates the [Azure confidential H100 AKS onboarding guide](https://github.com/Azure/az-cgpu-onboarding/blob/136262ef00ad75b8e0bc789563f043006fb68f4c/docs/Confidential-GPU-H100-AKS-Onboarding.md) +using **only its GPU Operator driver installation path**: +GPU Operator **v25.3.1**, open kernel modules, and +`nvcr.io/nvidia/driver:580--ubuntu24.04`. +There is no signed-package fallback, generic driver installation, manual host +kernel change, or Secure Boot bypass. + +## What "latest kernel" means + +The script selects the latest **AKS-managed node image offered to the new pool** +in the selected region, updates the pool if necessary, then reads the actual +worker kernel from Kubernetes. It requires `azure-fde` and a numeric version at +least **6.8.0-1071-azure-fde**. It does not select an arbitrary Ubuntu apt kernel: +AKS controls the kernel in its managed image. The default Kubernetes version is +the newest patch of the region's non-preview default minor; it is recorded with +the image versions. + +**Support boundary:** the linked guide explicitly validates Ubuntu 24.04.3 and +kernel `6.8.0-1041-azure-fde`. Following its installation method on a newer image +does not independently establish Microsoft/NVIDIA support for that combination. +The result always records `VendorSupportValidated: false`. If the exact branch +580 image is missing, the script records `BlockedMissingGuideDriver`, stops +the disposable cluster, and exits nonzero. It does not substitute another branch +or repeatedly allocate workers for the same missing image. + +## Run + +Prerequisites: PowerShell 7.2+, authenticated Azure CLI, kubectl, Helm 3, +OpenSSH `ssh-keygen`, registered `Ubuntu2404Preview`, and permissions/quota to +create AKS and its managed resources. Preflight checks the actual regional SKU +restrictions, NCC family quota, system-family quota and total core quota using +the included [preflight helper](Preflight.ps1). Quota does not guarantee capacity. +Run these commands from this sample directory. + +```powershell +.\Deploy.ps1 -SubscriptionId '' -PreflightOnly + +.\Deploy.ps1 -SubscriptionId '' ` + -ResourceGroup 'rg-aks-gpu-worker-unique' ` + -ClusterName 'aks-gpu-worker-unique' ` + -Location westeurope -StopExistingNcc +``` + +`-StopExistingNcc` explicitly authorizes stopping **all existing NCC compute in +the selected subscription and region**, including whole AKS clusters that own +NCC pools (their system nodes also stop). Standalone NCC VMs/VMSS are deallocated. +Non-NCC compute and other regions are not changed. Without the switch, running +NCC compute is an error; `-PreflightOnly` never stops or creates compute. +Other regions can be selected explicitly; West Europe is the default. + +Uses a fresh resource group, one `Standard_D4ds_v6` system node and one +`Standard_NCC40ads_H100_v5` worker (44 total vCPUs). Both pools use Ubuntu2404 +instead of the guide's older Ubuntu2204 system pool; the confidential GPU pool, +disabled automatic driver and GPU taint match the guide. Autoscaling is omitted +to stay within the single-GPU quota/cost envelope. Automatic image/Kubernetes +upgrades are disabled for reproducible testing. CLI extensions and kubeconfig +are isolated per run without changing shared CLI defaults or kubeconfig. + +After confirming ConfidentialVM, Secure Boot and vTPM, the script gates Helm on +a real registry manifest lookup. Authentication, rate-limit and network errors +are failures, not missing-image evidence. If installation can proceed, it +requires one allocatable GPU, CC ON/PRODUCTION/Ready, successful strict GPU +attestation, the guide's TensorFlow matrix multiplication job, and successful +strict attestation again. The TensorFlow workload retains the guide's image +and computation, but explicitly selects GPU 0, disables CPU fallback, and checks +the result tensor's actual device. Its runtime and retries are bounded, and it +uses the NVIDIA runtime on the GPU pool. The guide's attestation manifest is +pinned to the revision linked above; +it retains the guide's latest onboarding archive download, with strict OCSP, +bounded runtime and scheduling onto the GPU pool added. This is GPU attestation, +not a separate CPU/whole-VM attestation claim. + +The newly created cluster is stopped in `finally`, including after failure. +Bounded readiness waits allow AKS background addon/pool operations to settle +before subsequent provisioning, configuration checks and shutdown. +If an addon operation starts between readiness and shutdown, AKS stop retries +only `OperationNotAllowed` errors identifying an in-progress operation, for up +to 20 minutes. Retries emit warnings; other errors fail immediately. The same +shutdown helper is used for existing NCC AKS clusters. +Stop and VMSS instance deallocation are verified. `-KeepRunning` explicitly +disables this cost safeguard. Failed provisioning can prevent AKS stop; such +cleanup errors are surfaced and require immediate attention. Existing compute +stopped at the beginning is not automatically restarted. + +Run evidence is in ignored `.local/-/`: kernel, node image, +VMSS security, registry response, result and (when reached) workload/attestation +logs. Failed job pod status and redacted logs are captured before shutdown when +available; diagnostic collection errors are surfaced. Attestation tokens are +omitted from persisted logs. This directory also +contains a dedicated kubeconfig and SSH private key: **do not commit or share it**. +No resources are automatically deleted. Stopped AKS retains disks/IPs and other +resources that may still incur charges; delete its resource group when finished. + +## Offline validation + +```powershell +.\tests\Test-Sample.ps1 +``` + +Exercises numeric kernel floors, non-preview patch selection, genuine missing +images versus registry failures, digest validation, NCC-only shutdown scope, +explicit subscription propagation, real SSH key generation/error propagation, +guide workload GPU-placement guards and PowerShell parsing. +Also tests AKS background-operation readiness, failure and timeout handling. +Shutdown tests cover operation conflicts, nonretryable failures, deadline +exhaustion and explicit target/subscription propagation. Preflight tests check +read-only command scope, quota and registration failures, and saved evidence. + +## West Europe live validation + +On 2026-11-12, a fresh single-H100 deployment exited zero with +`Status: Succeeded`. It used Kubernetes +1.36.5, Ubuntu 24.04.5 LTS, the latest offered GPU image +`AKSUbuntu-2404gen2CVMcontainerd-202610.30.1`, kernel +`6.8.0-1071-azure-fde`, and the exact guide branch-580 driver image. +ConfidentialVM, Secure Boot, vTPM, CC ON/PRODUCTION/Ready and one allocatable +GPU checks passed. Strict GPU attestation passed before and after the workload. + +Actual TensorFlow output (informational/startup warnings omitted): + +```text +TensorFlow version: 2.15.0 +GPU Available: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')] +Running GPU matmul... +GPU matmul device: /job:localhost/replica:0/task:0/device:GPU:0 +Elapsed: 4.1502110958099365 sec +``` + +TensorFlow logged duplicate CUDA factory registration messages and a compute +capability 9.0 PTX JIT warning, but the job completed with explicit GPU placement. +Both redacted attestation logs reported `GPU Attestation is Successful.` for +the same worker GPU. + +Automatic shutdown completed in this run without manual recovery: +`FinalPowerState: Stopped`, `ComputeDeallocated: true`. An independent Azure +check at 14:43:14 UTC confirmed AKS Stopped/Succeeded and capacity zero / zero +instances in both the GPU and system scale sets. +The AKS identity's Contributor role was verified on exactly its managed node +resource group. +The vendor-support caveat above still applies; retained disks/IPs can still cost money. + +### Redacted GPU attestation example + +The verifier output below is from that successful run. GPU identifiers and +attestation tokens have been removed; package-installation noise is omitted. +Both the pre-workload and post-workload checks passed for the same GPU. +**Version numbers and exact output may change** as AKS images, drivers, firmware +and the upstream verifier evolve. This demonstrates GPU attestation only. + +```text +== Run attestation == +Generating random nonce in the local GPU Verifier .. +Number of GPUs available : 1 +Fetching GPU 0 information from GPU driver. +All GPU Evidences fetched successfully +----------------------------------- +Verifying GPU: + Driver version fetched : 580.178.04 + VBIOS version fetched : 96.00.9f.00.04 + Validating GPU certificate chains. + The firmware ID in the device certificate chain is matching with the one in the attestation report. + GPU attestation report certificate chain validation successful. + The certificate chain revocation status verification successful. + Authenticating attestation report + The nonce in the SPDM GET MEASUREMENT request message is matching with the generated nonce. + Driver version fetched from the attestation report : 580.178.04 + VBIOS version fetched from the attestation report : 96.00.9f.00.04 + Attestation report signature verification successful. + Attestation report verification successful. + Authenticating the RIMs. + Authenticating Driver RIM + Fetching the driver RIM from the RIM service. + RIM Schema validation passed. + driver RIM certificate chain verification successful. + The certificate chain revocation status verification successful. + driver RIM signature verification successful. + Driver RIM verification successful + Authenticating VBIOS RIM. + Fetching the VBIOS RIM from the RIM service. + RIM Schema validation passed. + vbios RIM certificate chain verification successful. + The certificate chain revocation status verification successful. + vbios RIM signature verification successful. + VBIOS RIM verification successful + Comparing measurements (runtime vs golden) + The runtime measurements are matching with the golden measurements. + GPU is in expected state. + GPU 0 with UUID verified successfully. + GPU Ready State is already READY +GPU Attestation is Successful. + +[Attestation token omitted] +``` diff --git a/aks-samples/aks-gpu-worker/tests/Test-Sample.ps1 b/aks-samples/aks-gpu-worker/tests/Test-Sample.ps1 new file mode 100644 index 0000000..0791590 --- /dev/null +++ b/aks-samples/aks-gpu-worker/tests/Test-Sample.ps1 @@ -0,0 +1,269 @@ +#Requires -Version 7.2 +. (Join-Path $PSScriptRoot '..\Common.ps1') +$script:passed = 0 + +function Assert-True { + param([bool]$Condition, [string]$Message) + if (-not $Condition) { throw $Message } + $script:passed++ +} + +function Assert-Throws { + param([scriptblock]$Action, [string]$Pattern) + $caught = $false + try { & $Action } catch { + if ($_.Exception.Message -notmatch $Pattern) { throw } + $caught = $true + } + Assert-True $caught "Expected error matching $Pattern." +} + +foreach ($kernel in @('6.8.0-1071-azure-fde', '6.8.0-1072-azure-fde', '6.8.0-11000-azure-fde', '6.11.0-1-azure-fde', '7.0.0-1-azure-fde')) { + Assert-MinimumKernel $kernel + $script:passed++ +} +foreach ($kernel in @('6.8.0-1070-azure-fde', '6.8.0-999-azure-fde', '6.7.0-9999-azure-fde')) { + Assert-Throws { Assert-MinimumKernel $kernel } 'older' +} +foreach ($kernel in @('6.8.0-1071-azure', '6.8.0-1071-generic', 'garbage')) { + Assert-Throws { Assert-MinimumKernel $kernel } 'Not an Azure FDE' +} +Assert-Throws { Assert-MinimumKernel '6.8.0-1071-azure-fde' 'garbage' } 'Invalid minimum' +$versions = @' +{"values":[ + {"isPreview":false,"patchVersions":{"1.36.9":{},"1.36.10":{}}}, + {"isPreview":true,"patchVersions":{"1.99.0":{}}}, + {"isPreview":null,"patchVersions":{"1.35.11":{}}} +]} +'@ | ConvertFrom-Json +Assert-True ((Get-LatestKubernetesPatch $versions) -eq '1.36.10') 'Patch selection must be numerical and exclude preview.' +Assert-Throws { Get-LatestKubernetesPatch ([pscustomobject]@{ values = @() }) } 'No stable' + +$keyDirectory = Join-Path ([System.IO.Path]::GetTempPath()) "aks-gpu-key-test-$([guid]::NewGuid())" +New-Item -ItemType Directory -Path $keyDirectory | Out-Null +$keyPath = Join-Path $keyDirectory 'ssh-key' +try { + New-SampleSshKey -Path $keyPath + Assert-True ((Get-Content "$keyPath.pub" -Raw) -match '^ssh-rsa ') 'Expected a real RSA public key.' + Assert-Throws { New-SampleSshKey -Path (Join-Path $keyDirectory 'missing\ssh-key') } 'ssh-keygen failed' +} +finally { + foreach ($path in @($keyPath, "$keyPath.pub")) { + if (Test-Path $path) { Remove-Item -LiteralPath $path -Force } + } + Remove-Item -LiteralPath $keyDirectory +} + +function Invoke-RestMethod { param($Uri) return @{ token = 'unit-test-token' } } +function Invoke-WebRequest { + param($Uri, $Headers, [switch]$SkipHttpErrorCheck) + Assert-True ($Headers.Authorization -eq 'Bearer unit-test-token') 'Registry authentication was not passed.' + Assert-True ($Uri -like '*/580-6.8.0-1071-azure-fde-ubuntu24.04') 'Driver tag differs from the guide.' + return $script:response +} +$script:response = @{ + StatusCode = 404 + Content = '{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown"}]}' +} +$image = Get-ExactDriverImages '6.8.0-1071-azure-fde' @('580') +Assert-True (-not $image.Available -and $image.HttpStatus -eq 404 -and -not $image.Digest) 'Missing-image evidence was incorrect.' +foreach ($status in @(401, 403, 429, 500)) { + $script:response = @{ StatusCode = $status } + Assert-Throws { Get-ExactDriverImages '6.8.0-1071-azure-fde' @('580') } "HTTP $status" +} +$script:response = @{ StatusCode = 404; Content = '{"errors":[{"code":"NAME_UNKNOWN","message":"repository unknown"}]}' } +Assert-Throws { Get-ExactDriverImages '6.8.0-1071-azure-fde' @('580') } 'Unexpected registry 404' +$script:response = @{ StatusCode = 200; Headers = @{ 'Docker-Content-Digest' = @('sha256:' + ('a' * 64)) } } +$image = Get-ExactDriverImages '6.8.0-1071-azure-fde' @('580') +Assert-True ($image.Available -and $image.Digest -eq ('sha256:' + ('a' * 64))) 'Available-image evidence was incorrect.' +$script:response = @{ StatusCode = 200; Headers = @{ 'Docker-Content-Digest' = @('not-a-digest') } } +Assert-Throws { Get-ExactDriverImages '6.8.0-1071-azure-fde' @('580') } 'Missing registry digest' +Remove-Item Function:\Invoke-RestMethod +Remove-Item Function:\Invoke-WebRequest + +$workload = @' +apiVersion: batch/v1 +kind: Job +metadata: + name: tf-matmul-demo +spec: + template: + spec: + restartPolicy: OnFailure + containers: + - name: tf-matmul-demo + image: tensorflow/tensorflow:2.15.0-gpu + args: + - | + c = tf.matmul(a, b) +'@ +$guardedWorkload = Get-GuideWorkloadManifest $workload +Assert-True ($guardedWorkload.Contains('activeDeadlineSeconds: 900')) 'Workload deadline missing.' +Assert-True ($guardedWorkload.Contains('restartPolicy: Never')) 'Workload should not retry indefinitely.' +Assert-True ($guardedWorkload.Contains('runtimeClassName: nvidia')) 'NVIDIA runtime missing.' +Assert-True ($guardedWorkload.Contains('agentpool: gpunp')) 'Workload must target GPU pool.' +Assert-True ($guardedWorkload.Contains('tf.config.set_soft_device_placement(False)')) 'CPU fallback must be disabled.' +Assert-True ($guardedWorkload.Contains("with tf.device('/GPU:0'):")) 'Matmul must explicitly target GPU.' +Assert-True ($guardedWorkload.Contains("assert 'GPU:0' in c.device")) 'Actual tensor device must be checked.' +Assert-True ($guardedWorkload.Contains('image: tensorflow/tensorflow:2.15.0-gpu')) 'Guide workload image changed.' +Assert-Throws { Get-GuideWorkloadManifest ($workload.Replace('c = tf.matmul(a, b)', 'unexpected')) } 'Upstream workload manifest changed' + +$script:commands = [System.Collections.Generic.List[string]]::new() +function Invoke-CheckedCommand { + param([string]$Command, [string[]]$ArgumentList) + $call = $ArgumentList -join ' ' + $script:commands.Add($call) + if ($call -like 'aks list *') { return '[]' } + if ($call -like 'vmss list *') { return '[]' } + if ($call -like 'vm list *') { + return '[{"name":"ncc","location":"westeurope","powerState":"VM running","id":"/vm/ncc","hardwareProfile":{"vmSize":"Standard_NCC40ads_H100_v5"}},{"name":"untouched","location":"westeurope","powerState":"VM running","hardwareProfile":{"vmSize":"Standard_D2s_v5"}}]' + } + if ($call -like 'vm deallocate *') { return } + if ($call -like 'vm get-instance-view *') { return $script:power } + throw "Unexpected test command: $call" +} +Assert-Throws { Stop-ExistingNccCompute -Az az -Subscription test -Location westeurope } 'needs -StopExistingNcc' +Assert-True (-not @($script:commands | Where-Object { $_ -like '* deallocate *' }).Count) 'Read-only inventory modified compute.' +$script:commands.Clear() +$script:power = 'PowerState/deallocated' +Stop-ExistingNccCompute -Az az -Subscription test -Location westeurope -Stop +Assert-True (@($script:commands | Where-Object { $_ -like 'vm deallocate *' }).Count -eq 1) 'Expected exactly one NCC shutdown.' +Assert-True (-not @($script:commands | Where-Object { $_ -like '*untouched*' }).Count) 'Non-NCC VM was modified.' +Assert-True (-not @($script:commands | Where-Object { $_ -notlike '*--subscription test*' }).Count) 'Missing subscription scope.' +$script:power = 'PowerState/running' +Assert-Throws { Stop-ExistingNccCompute -Az az -Subscription test -Location westeurope -Stop } 'did not deallocate' +Remove-Item Function:\Invoke-CheckedCommand + +$script:readCount = 0 +$script:readyStates = @('Updating', 'Succeeded') +function Invoke-CheckedCommand { + param([string]$Command, [string[]]$ArgumentList) + Assert-True ('--subscription' -in $ArgumentList -and 'test' -in $ArgumentList) 'Readiness lookup lost subscription scope.' + $state = $script:readyStates[[Math]::Min($script:readCount, $script:readyStates.Count - 1)] + if ($ArgumentList[1] -eq 'show') { + return (@{ provisioningState = $state } | ConvertTo-Json) + } + $script:readCount++ + return "[{`"name`":`"sysnp`",`"provisioningState`":`"$state`"}]" +} +Wait-SampleAksReady az test group cluster -PollSeconds 0 +Assert-True ($script:readCount -eq 2) 'Readiness must wait for background AKS operations.' +$script:readyStates = @('Failed') +Assert-Throws { Wait-SampleAksReady az test group cluster -PollSeconds 0 } 'provisioning failed' +$script:readyStates = @('Updating') +Assert-Throws { Wait-SampleAksReady az test group cluster -TimeoutSeconds 0 -PollSeconds 0 } 'did not settle' +Remove-Item Function:\Invoke-CheckedCommand + +$script:stopCalls = 0 +$script:stopFailures = 1 +$script:stopError = '(OperationNotAllowed) There is an in-progress PutExtensionAddonHandler.PUT operation.' +function Invoke-TestAksStop { + Assert-True (($args -join ' ') -eq 'aks stop -g group -n cluster -o none --subscription test --only-show-errors') 'Shutdown lost explicit target/subscription scope.' + $script:stopCalls++ + $global:LASTEXITCODE = 0 + if ($script:stopCalls -le $script:stopFailures) { + $global:LASTEXITCODE = 1 + $script:stopError + } +} +Stop-SampleAks Invoke-TestAksStop test group cluster -PollSeconds 0 +Assert-True ($script:stopCalls -eq 2) 'In-progress operation must be retried.' +foreach ($errorMessage in @('(AuthorizationFailed) Denied', + '(OperationNotAllowed) Unsupported cluster configuration', + 'There is an in-progress operation, but a different error code')) { + $script:stopCalls = 0 + $script:stopError = $errorMessage + Assert-Throws { Stop-SampleAks Invoke-TestAksStop test group cluster -PollSeconds 0 } 'AKS stop failed' + Assert-True ($script:stopCalls -eq 1) 'Nonretryable shutdown failure was retried.' +} +$script:stopCalls = 0 +$script:stopError = '(OperationNotAllowed) There is an in-progress PutExtensionAddonHandler.PUT operation.' +Assert-Throws { Stop-SampleAks Invoke-TestAksStop test group cluster -TimeoutSeconds 0 -PollSeconds 0 } 'remained blocked' +Assert-True ($script:stopCalls -eq 1) 'Expired shutdown deadline must stop retrying.' +Remove-Item Function:\Invoke-TestAksStop +$global:LASTEXITCODE = 0 + +$script:preflightCommands = [System.Collections.Generic.List[string]]::new() +$script:accountState = 'Enabled' +$script:featureState = 'Registered' +$script:familyLimit = 40 +$script:regionalLimit = 44 +$script:restricted = $false +function Invoke-CheckedCommand { + param([string]$Command, [string[]]$ArgumentList) + $call = $ArgumentList -join ' ' + $script:preflightCommands.Add($call) + if ($call -eq 'version -o json') { return '{"azure-cli":"2.92.0"}' } + if ($call -notlike '*--subscription test*') { throw 'Preflight lost subscription scope.' } + if ($call -like 'account show *') { return "{`"state`":`"$script:accountState`"}" } + if ($call -like 'aks get-versions *') { + return '{"values":[{"patchVersions":{"1.36.5":{}}}]}' + } + if ($call -like 'vm list-skus *') { + $restrictions = @(if ($script:restricted) { @{ type = 'Location' } }) + return (ConvertTo-Json -Depth 10 -InputObject @( + @{ name = 'Standard_D4ds_v6'; family = 'system'; restrictions = @(); capabilities = @(@{ name = 'vCPUs'; value = '4' }) } + @{ name = 'Standard_NCC40ads_H100_v5'; family = 'ncc'; restrictions = $restrictions; capabilities = @(@{ name = 'vCPUs'; value = '40' }) } + )) + } + if ($call -like 'vm list-usage *') { + return (ConvertTo-Json -Depth 10 -InputObject @( + @{ name = @{ value = 'system' }; currentValue = 0; limit = 4 } + @{ name = @{ value = 'ncc' }; currentValue = 0; limit = $script:familyLimit } + @{ name = @{ value = 'cores' }; currentValue = 0; limit = $script:regionalLimit } + )) + } + if ($call -like 'feature show *') { return $script:featureState } + throw "Unexpected preflight command: $call" +} +$preflightDirectory = Join-Path ([System.IO.Path]::GetTempPath()) "aks-gpu-preflight-test-$([guid]::NewGuid())" +try { + Test-SamplePreflight az test westeurope '1.36.5' $preflightDirectory + $evidencePath = Join-Path $preflightDirectory 'preflight.json' + $evidence = Get-Content $evidencePath -Raw | ConvertFrom-Json + Assert-True ($evidence.Skus.Count -eq 2 -and $evidence.Quota.Count -eq 3) 'Preflight evidence must include both SKUs and all quotas.' + Assert-True ($evidence.Location -eq 'westeurope' -and $evidence.PreviewState -eq 'Registered') 'Preflight evidence lost location or registration.' + Assert-True (-not @($script:preflightCommands | Where-Object { $_ -notmatch '^(account show|version|aks get-versions|vm list-skus|vm list-usage|feature show) ' }).Count) 'Preflight modified Azure resources.' + Assert-Throws { Test-SamplePreflight az test westeurope '1.99.0' $preflightDirectory } 'not offered' + $script:accountState = 'Disabled' + Assert-Throws { Test-SamplePreflight az test westeurope '1.36.5' $preflightDirectory } 'not enabled' + $script:accountState = 'Enabled' + $script:familyLimit = 39 + Assert-Throws { Test-SamplePreflight az test westeurope '1.36.5' $preflightDirectory } 'Insufficient ncc quota' + $script:familyLimit = 40 + $script:regionalLimit = 43 + Assert-Throws { Test-SamplePreflight az test westeurope '1.36.5' $preflightDirectory } 'Insufficient regional' + $script:regionalLimit = 44 + $script:restricted = $true + Assert-Throws { Test-SamplePreflight az test westeurope '1.36.5' $preflightDirectory } 'regional subscription restriction' + $script:restricted = $false + $script:featureState = 'NotRegistered' + Assert-Throws { Test-SamplePreflight az test westeurope '1.36.5' $preflightDirectory } 'Arrange subscription registration' +} +finally { + if (Test-Path $preflightDirectory) { + Remove-Item -LiteralPath (Join-Path $preflightDirectory 'preflight.json') -Force + Remove-Item -LiteralPath $preflightDirectory + } + Remove-Item Function:\Invoke-CheckedCommand +} + +$root = Split-Path $PSScriptRoot +foreach ($file in Get-ChildItem $root -Filter '*.ps1' -Recurse | Where-Object FullName -NotMatch '[\\/]\.local[\\/]') { + $tokens = $null + $errors = $null + [System.Management.Automation.Language.Parser]::ParseFile($file.FullName, [ref]$tokens, [ref]$errors) | Out-Null + Assert-True ($errors.Count -eq 0) "Syntax error in $($file.Name): $errors" + if ($file.DirectoryName -eq $root) { + Assert-True ((Get-Content $file.FullName -Raw) -notmatch 'aks-gpu-worker-temp|\.\.\\aks-gpu-worker\\') "Stale sibling dependency in $($file.Name)." + } +} +$source = Get-Content (Join-Path $root 'Deploy.ps1') -Raw +Assert-True ($source.IndexOf('BlockedMissingGuideDriver') -lt $source.IndexOf("helm @('install'")) 'Missing-driver gate must precede Helm installation.' +Assert-True ($source -match "-Branches @\('580'\)") 'Only the guide driver branch is allowed.' +Assert-True ($source -notmatch 'Install-SignedDriver|driver.enabled=false|apt-get.*linux') 'An alternative host-driver installation was introduced.' +$readme = Get-Content (Join-Path $root 'README.md') -Raw +Assert-True ($readme -notmatch 'GPU-[0-9a-f]{8}-|aks-gpu-worker-temp|aks-gpunp-\d+|aks-sysnp-\d+') 'README contains deployment identifiers or stale names.' +Assert-True ($readme.Contains('Version numbers and exact output may change')) 'Attestation version caveat missing.' +Assert-True ($readme.Contains('GPU Attestation is Successful.') -and $readme.Contains('[Attestation token omitted]')) 'Redacted attestation example missing.' +Write-Host "Passed $script:passed assertions."