diff --git a/doc/ci.md b/doc/ci.md index 770014d4e..7493f27b1 100644 --- a/doc/ci.md +++ b/doc/ci.md @@ -217,6 +217,26 @@ mount the share, read the file that the host wrote, and write one that the host then reads, and the restore may not change the snapshot. Until OpenVMM's virtio-fs queue workers waited for the gate to release (#904), the clone's mount succeeded but its first read never returned. +The `platform-aggregate-export` scenario covers an aggregate of host +directories and files that a platform restore attaches to the filesystem slot +that the capture left dormant, as a clone's export does. It captures a +platform snapshot without a filesystem and restores it with the aggregate of +the `filesystem-shares` scenario: a read-write workspace and a read-only tool +cache, each with a denied path, a read-write data directory whose root exposes +only its allowed paths and whose notes alone are writable, and a read-only and +a read-write file. The clone keeps its captured command line, which names no +filesystem, so its guest must mount the aggregate itself, find the five +children in its read-only root, which only root can enter, check each child's +kind, mode, and policy, and bind a directory and a file child, and the host +must then find exactly the writes that the policy admits. Before the guest +runs, OpenVMM must refuse the restores whose aggregates a cold boot refuses, +with overlapping directories, a file inside a directory child, a child of the +wrong kind, a repeated name, a policy path of a file child, one outside every +child, or a relative one, and on a Windows host caller ownership, and no +restore may change the snapshot. `filesystem-snapshot` also requires OpenVMM +to refuse an aggregate for the dormant slot of its untiered snapshot, which +binds only a `--mount`, because only a platform snapshot's dormant slot binds +an aggregate. The `filesystem-snapshot` scenario's blockless capture of a live share covers the write-back of shared mappings (#555). A guest process dirties a page of a file in the share through a shared mapping with `nvx-device-io map` and holds diff --git a/doc/openvmm-upstream-roadmap.md b/doc/openvmm-upstream-roadmap.md index 10bb29f74..788d04b9a 100644 --- a/doc/openvmm-upstream-roadmap.md +++ b/doc/openvmm-upstream-roadmap.md @@ -9,19 +9,20 @@ - Mapped fork tip: `a0556d7d85ce1fff89f5b45a98f15c5c79cbb6fb`, position 157. The graph, PR summary, and commit mapping below cover positions 001-157 and the fixups at positions 334-340, 344, and 345. -- NVX's pinned OpenVMM revision: `b92daa350c93914b2eb556acf47886cc313d396b`, position 352. - Positions 350-352, which microsoft/openvmm#4773 proposes for `feature/microvm/dev`, are - three commits on position 349, `d2338ea22`, the head of `feature/microvm/dev` since - microsoft/openvmm#4769 merged positions 348 and 349. Those are two commits on position 347, - `57d0a18b3`, which microsoft/openvmm#4767 merged with position 346. Positions 346 and 347 are - two commits on position 345, `e68871cc6`, which microsoft/openvmm#4762 merged with position - 344. Positions 344 and 345 are two commits on position 343, `a12d9167f`, which - microsoft/openvmm#4749 merged with positions 341 and 342. Positions 341-343 are three commits - on position 340, `2b681564d`, which microsoft/openvmm#4735 merged with position 339. - Positions 339 and 340 are two commits on position 338, `8a0d945ed`, which - microsoft/openvmm#4734 merged, and position 338 is one commit on `79a501b63`, the head of the - `feature/microvm/release/v2.3` branch of microsoft/openvmm. The pin is 195 commits past the - mapped tip on the same first-parent chain. Positions 158-333, 341-343, and 346-352 are not +- NVX's pinned OpenVMM revision: `832d5bb48dadd2cc5221f2c8b8905e6b80d13d53`, position 354. + Positions 353 and 354, which microsoft/openvmm#4775 proposes for `feature/microvm/dev`, are + two commits on position 352, `b92daa350`, the head of `feature/microvm/dev` since + microsoft/openvmm#4773 merged positions 350-352. Those are three commits on position 349, + `d2338ea22`, which microsoft/openvmm#4769 merged with position 348. Positions 348 and 349 are + two commits on position 347, `57d0a18b3`, which microsoft/openvmm#4767 merged with position + 346. Positions 346 and 347 are two commits on position 345, `e68871cc6`, which + microsoft/openvmm#4762 merged with position 344. Positions 344 and 345 are two commits on + position 343, `a12d9167f`, which microsoft/openvmm#4749 merged with positions 341 and 342. + Positions 341-343 are three commits on position 340, `2b681564d`, which microsoft/openvmm#4735 + merged with position 339. Positions 339 and 340 are two commits on position 338, `8a0d945ed`, + which microsoft/openvmm#4734 merged, and position 338 is one commit on `79a501b63`, the head + of the `feature/microvm/release/v2.3` branch of microsoft/openvmm. The pin is 197 commits past + the mapped tip on the same first-parent chain. Positions 158-333, 341-343, and 346-354 are not yet assigned to PRs; see [Commits after the mapped tip](#commits-after-the-mapped-tip). - Upstream main: `22c0485be597f87a76771e1db03eaa823a685ffc` (2026-10-22), the fork base. @@ -312,8 +313,8 @@ The physical branch is one first-parent chain: -> PR14 fixup [336] -> PR09 fixup [337] -> PR02 fixup [338] -> PR20 fixup [339] -> PR23 fixup [340] -> unassigned [341-343] -> PR18 fixups [344-345] - -> unassigned [346-352] - -> b92daa350 (NVX pin) + -> unassigned [346-354] + -> 832d5bb48 (NVX pin) ``` For upstream review, PR01-PR08 and PR14 can be rebased independently onto current upstream main, @@ -335,8 +336,8 @@ PR18's position 111. ## Commits after the mapped tip -The NVX pin extends the chain past position 157 without merges. Of the 195 later commits, -positions 158-333, 341-343, and 346-352 are not yet classified or assigned to PRs, positions +The NVX pin extends the chain past position 157 without merges. Of the 197 later commits, +positions 158-333, 341-343, and 346-354 are not yet classified or assigned to PRs, positions 334-337 are the fixups of the `feature/microvm/release/v2.3` rebase, position 338 is a PR02 fixup, positions 339 and 340 are PR20 and PR23 fixups, and positions 344 and 345 are PR18 fixups. @@ -386,6 +387,8 @@ This table groups them by topic in chain order; ranges are inclusive, as in the | 347 | `57d0a18b3` | 1 | `--restore-network-policy` lets a platform restore install the policy of its own options on a NIC template, including private-network ingress, explicit host loopback, and forwards, before any vCPU runs, while the NIC keeps its identity and the contract keeps the template's deny-all (microsoft/openvmm#4767) | | 348-349 | `689577604..d2338ea22` | 2 | A second revision of the Genoa CPU profile, `amd.genoa.v2`, which `auto` selects: `amd.genoa.v1` without the basic leaves above `0xd` and without `LFENCE` serialization, NoSmmCtlMsr, and the TSA immunities in `0x80000021`, none of which the Microsoft hypervisor of the Azure Genoa VMs fingerprinted so far presents, with the nested-WHP Genoa host of microsoft/nvx#928 as a test fixture (microsoft/openvmm#4769); revises position 282's Genoa profile | | 350-352 | `8c1772d83..b92daa350` | 3 | HTTP blob disks bound each request in time, retry the ones that fail transiently on a new connection, and show them in inspect, so that a network path that silently stops delivering a connection's packets no longer hangs a guest's disk read (microsoft/openvmm#4773) | +| 353 | `057a2eaa6` | 1 | Test that the microVM virtio-fs device takes a dormant slot's state into the device of an aggregate of a directory and a file, which then saves the aggregate | +| 354 | `832d5bb48` | 1 | A platform restore may bind an aggregate, `--mount-aggregate` with its children, policy paths, and owner mode, to the filesystem slot that the snapshot left dormant, validated as at a cold boot, while the slot's device and the captured command line stay as captured; other tiers keep binding only a `--mount` (microsoft/openvmm#4775) | Some of these commits change mapped work rather than add to it. Positions 246-250 remove the legacy microVM clock paths and change files that PR10 and PR17 also change. Position 252 makes @@ -393,6 +396,8 @@ manifest version 6 the only snapshot format and changes files that PR11 and PR17 Positions 342 and 343 change the sandbox block attachment of PR19's position 115 and the restore of positions 285 and 286. Positions 346 and 347 change the snapshot network policy and the forward refusal of PR21's position 129 and the snapshot refusals of positions 306 and 307. +Position 354 changes the dormant-slot restore of PR18's position 113, as the aggregates of +position 293 restricted it. Assigning the later commits therefore means revisiting those PRs, not only appending new ones. Positions 350-352, by contrast, change only `disk_blob`, which upstream main and the fork base have alike and no other commit of the chain changes, so they can go upstream on their own. diff --git a/doc/run.md b/doc/run.md index 8106a3315..4b2c32b09 100644 --- a/doc/run.md +++ b/doc/run.md @@ -573,6 +573,39 @@ without a NIC, and for a NIC captured with any other policy, and every other [networked restore](#networked-snapshots) requires the captured policy. `nvx-rs run` does not pass the option. +### Platform filesystem exports + +A platform snapshot captured without `--mount` or `--mount-aggregate` leaves +the virtio-fs slot dormant. Each restore may bind one directory to it with +`--mount`, or an aggregate of directories and files with `--mount-aggregate` +and its `--mount-child` options, for example: + +```bash +./bin/openvmm \ + --single-process \ + --machine microvm \ + --hypervisor kvm \ + --restore-snapshot /var/lib/nvx/template \ + --microvm-sandbox-block distro:file:/var/lib/nvx/image.erofs,ro \ + --restore-absent-scratch \ + --mount-aggregate /run/nvx/shares \ + --mount-child 0,/srv/checkout,rw \ + --mount-child 1,/opt/hostedtoolcache,ro \ + --mount-child 2,/srv/app/config/settings.json,ro,file \ + --mount-deny /srv/checkout/.git/credentials +``` + +OpenVMM validates the aggregate's children, their `--mount-deny`, +`--mount-allow`, and `--mount-write` paths, and the `--mount-owner` mode as it +validates a cold boot's, and attaches the aggregate before any vCPU runs. The +slot's device and the guest's kernel command line stay as captured, without a +`virtfs_*` or `nvx_share=` token, so the guest must learn of the export from +its host, mount tag `microvm` itself, and bind each child where it needs it. +The dormant slot of a snapshot of another tier, or without a tier, still +binds only a `--mount`, and a snapshot captured with an export requires that +export again. `nvx-rs run`, which passes no sandbox blocks, cannot restore a +platform snapshot. + ## Migration from the retired profile `microvm` is now the only selector and launches the contract previously named @@ -747,7 +780,10 @@ Capture and restore inspect and reopen the guest's open files as OpenVMM's user, so unless OpenVMM runs as root, they fail while the guest holds a file that only its caller can reach or reopen, such as one open for writing. A snapshot captured without a mapping may restore with one new `--mount` of a -directory, but not with several or with a file; after resume, mount it +directory, but not with several or with a file, because `nvx-rs run` restores +only snapshots without sandbox blocks, whose dormant slot takes no aggregate; +OpenVMM binds an aggregate to the dormant slot of a +[platform snapshot](#platform-filesystem-exports). After resume, mount it explicitly inside the guest because the initramfs hook has already completed: ```sh diff --git a/doc/usage.md b/doc/usage.md index 5e478f299..833361e73 100644 --- a/doc/usage.md +++ b/doc/usage.md @@ -1104,9 +1104,9 @@ omit. Alpine remains the default. Ubuntu and Azure Linux cannot act as sandbox control, so they reject the Alpine-control-only `sandbox-blocks` and `scratch-snapshot` scenarios and the sandbox-control-dependent `snapshot-tiers`, `platform-restore-processors`, -`platform-zero-fill`, `platform-filesystem`, `platform-network-policy`, -`managed-exec-config`, `sandbox-lifecycle`, and `sandbox-proxy-only` -scenarios. Ubuntu also rejects +`platform-zero-fill`, `platform-filesystem`, `platform-aggregate-export`, +`platform-network-policy`, `managed-exec-config`, `sandbox-lifecycle`, and +`sandbox-proxy-only` scenarios. Ubuntu also rejects the Alpine-prompt-specific `console-snapshot` scenario. `--debug-kernel` boots the CI debug kernel, `build/vmlinux-debug`, whose soft-lockup and hung-task detectors the guest's time ABI watcher reports; without `--scenario`, it runs diff --git a/openvmm b/openvmm index b92daa350..832d5bb48 160000 --- a/openvmm +++ b/openvmm @@ -1 +1 @@ -Subproject commit b92daa350c93914b2eb556acf47886cc313d396b +Subproject commit 832d5bb48dadd2cc5221f2c8b8905e6b80d13d53 diff --git a/scripts/nvx_tools/microvm_shards.json b/scripts/nvx_tools/microvm_shards.json index e84e89773..39d15771e 100644 --- a/scripts/nvx_tools/microvm_shards.json +++ b/scripts/nvx_tools/microvm_shards.json @@ -46,6 +46,7 @@ "lifecycle": 1.7, "managed-lifecycle": 4.2, "network-snapshot": 17.0, + "platform-aggregate-export": 3.4, "platform-filesystem": 6.0, "platform-network-policy": 13.7, "platform-restore-processors": 16.0, @@ -67,8 +68,9 @@ }, "shards": [ { - "seconds": 249.9, + "seconds": 251.4, "scenarios": [ + "command-line-limit", "console-snapshot", "directional-network-policy", "endpoint-policy-snapshot", @@ -87,9 +89,8 @@ ] }, { - "seconds": 249.5, + "seconds": 251.4, "scenarios": [ - "command-line-limit", "console-exit", "cpu-profile-fallback", "denied-filesystem-paths", @@ -104,6 +105,7 @@ "l3-l4-egress-port-ranges", "l3-l4-egress-protocols", "lifecycle", + "platform-aggregate-export", "platform-filesystem", "platform-network-policy", "platform-zero-fill", @@ -142,6 +144,7 @@ "lifecycle": 1.6, "managed-lifecycle": 4.3, "network-snapshot": 16.8, + "platform-aggregate-export": 4.5, "platform-filesystem": 6.0, "platform-network-policy": 16.5, "platform-restore-processors": 16.0, @@ -163,7 +166,7 @@ }, "shards": [ { - "seconds": 230.5, + "seconds": 232.5, "scenarios": [ "console-snapshot", "directional-network-policy", @@ -173,8 +176,8 @@ "network-snapshot", "platform-filesystem", "platform-restore-processors", + "platform-zero-fill", "restore-downtime", - "restore-memory", "restore-processors", "scratch-snapshot", "smp", @@ -184,7 +187,7 @@ ] }, { - "seconds": 230.0, + "seconds": 232.5, "scenarios": [ "command-line-limit", "console-exit", @@ -202,9 +205,10 @@ "l3-l4-egress-protocols", "lifecycle", "managed-lifecycle", + "platform-aggregate-export", "platform-network-policy", - "platform-zero-fill", "private-network-ingress", + "restore-memory", "sandbox-blocks", "structured-outcome", "time-abi-conformance", @@ -238,6 +242,7 @@ "lifecycle": 2.1, "managed-lifecycle": 17.8, "network-snapshot": 17.6, + "platform-aggregate-export": 9.1, "platform-filesystem": 6.0, "platform-network-policy": 24.8, "platform-restore-processors": 16.0, @@ -259,7 +264,7 @@ }, "shards": [ { - "seconds": 327.1, + "seconds": 331.7, "scenarios": [ "console-snapshot", "denied-filesystem-paths", @@ -268,6 +273,7 @@ "filesystem-snapshot", "host-loopback-policy", "network-snapshot", + "platform-aggregate-export", "platform-filesystem", "platform-restore-processors", "restore-downtime", @@ -276,12 +282,11 @@ "scratch-snapshot", "smp", "smp-snapshot", - "snapshot-tiers", - "virtio-net" + "snapshot-tiers" ] }, { - "seconds": 327.2, + "seconds": 331.7, "scenarios": [ "command-line-limit", "console-exit", @@ -305,6 +310,7 @@ "snapshot-core", "structured-outcome", "time-abi-conformance", + "virtio-net", "workload-identity" ] } diff --git a/scripts/nvx_tools/microvm_test_scripts/platform-aggregate-export.sh b/scripts/nvx_tools/microvm_test_scripts/platform-aggregate-export.sh new file mode 100644 index 000000000..928d67082 --- /dev/null +++ b/scripts/nvx_tools/microvm_test_scripts/platform-aggregate-export.sh @@ -0,0 +1,85 @@ +set -eu +on_exit() { + status=$? + if [ "$status" -ne 0 ]; then + nvx-exit "$status" + fi +} +trap on_exit EXIT +fail() { + echo "NVX-PLATFORM-AGGREGATE-FAIL $1" + exit "$2" +} +expect_error() { + case "$1" in + *"$2"*) ;; + *) fail "$3: $1" "$4" ;; + esac +} +# The restore attached an aggregate to the slot that the platform capture left +# dormant. The guest keeps its captured command line, which discovers the slot +# but names no filesystem, so the guest mounts the aggregate itself. +grep -q 'virtio_mmio.device=0x1000@0xd0001000:6' /proc/cmdline || fail "discovery" 80 +if grep -Eq '(^| )(virtfs_|nvx_share=)' /proc/cmdline; then + fail "command line" 81 +fi +shares=/run/nvx/shares +mkdir -p "$shares" +mount -t virtiofs microvm "$shares" || fail "mount" 82 +[ "$(ls "$shares" | tr '\n' ' ')" = "0 1 2 3 4 " ] || fail "children" 83 +[ "$(stat -c %a "$shares")" = 500 ] || fail "root mode" 84 + +# The read-write workspace hides its denied path and accepts writes. +[ "$(cat "$shares/0/seed")" = NVX-WORKSPACE ] || fail "workspace read" 85 +[ ! -e "$shares/0/secrets" ] || fail "workspace denied path" 86 +printf 'NVX-GUEST-WRITE\n' >"$shares/0/from-guest" || fail "workspace write" 87 +mkdir "$shares/0/guest-directory" || fail "workspace directory" 88 + +# The read-only tool cache hides its denied path and rejects writes. +[ "$(cat "$shares/1/tools/node")" = NVX-TOOL ] || fail "tool cache read" 89 +[ ! -e "$shares/1/credentials" ] || fail "tool cache denied path" 90 +if error=$(touch "$shares/1/mutation" 2>&1); then + fail "tool cache write" 91 +fi +expect_error "$error" 'Read-only file system' "tool cache write" 92 + +# The data directory's root exposes only its allowed paths, and only the notes +# are writable. +[ "$(ls "$shares/2" | tr '\n' '|')" = "notes one.txt|public|" ] || fail "data listing" 93 +[ ! -e "$shares/2/private.txt" ] || fail "data hidden path" 94 +[ "$(cat "$shares/2/public/readme")" = NVX-PUBLIC ] || fail "data allowed path" 95 +printf 'NVX-GUEST-NOTE\n' >>"$shares/2/notes one.txt" || fail "data writable path" 96 +if error=$(touch "$shares/2/public/mutation" 2>&1); then + fail "data read-only path" 97 +fi +expect_error "$error" 'Read-only file system' "data read-only path" 98 +if error=$(touch "$shares/2/added" 2>&1); then + fail "data hidden root" 99 +fi +expect_error "$error" 'Permission denied' "data hidden root" 100 + +# Each file child is the file alone, with its own mode. +[ -f "$shares/3" ] && [ -f "$shares/4" ] || fail "file children" 101 +[ "$(cat "$shares/3")" = NVX-SETTINGS ] || fail "settings read" 102 +if error=$(ls "$shares/3/secret.json" 2>&1); then + fail "settings directory" 103 +fi +expect_error "$error" 'Not a directory' "settings directory" 104 +if error=$( (printf 'append\n' >>"$shares/3") 2>&1); then + fail "settings write" 105 +fi +expect_error "$error" 'Read-only file system' "settings write" 106 +printf 'NVX-GUEST-OUTPUT\n' >"$shares/4" || fail "output write" 107 +if error=$(rm -f "$shares/4" 2>&1); then + fail "output removal" 108 +fi +expect_error "$error" 'Read-only file system' "output removal" 109 + +# The guest binds the children where it needs them, a file child at a file. +mkdir -p /workspace /config +mount --bind "$shares/0" /workspace || fail "workspace bind" 110 +[ "$(cat /workspace/from-guest)" = NVX-GUEST-WRITE ] || fail "workspace bind read" 111 +: >/config/settings.json +mount --bind "$shares/3" /config/settings.json || fail "settings bind" 112 +[ "$(cat /config/settings.json)" = NVX-SETTINGS ] || fail "settings bind read" 113 +echo NVX-PLATFORM-AGGREGATE-OK diff --git a/scripts/nvx_tools/microvm_tests.py b/scripts/nvx_tools/microvm_tests.py index c43fa965a..a2e9a2329 100644 --- a/scripts/nvx_tools/microvm_tests.py +++ b/scripts/nvx_tools/microvm_tests.py @@ -119,6 +119,7 @@ "managed-lifecycle", "managed-exec-config", "network-snapshot", + "platform-aggregate-export", "platform-filesystem", "platform-network-policy", "platform-restore-processors", @@ -144,6 +145,7 @@ SANDBOX_CONTROL_SCENARIOS = frozenset( ( "managed-exec-config", + "platform-aggregate-export", "platform-filesystem", "platform-network-policy", "platform-restore-processors", @@ -387,6 +389,38 @@ PLATFORM_FILESYSTEM_MARKER = b"NVX-PLATFORM-FILESYSTEM-OK" PLATFORM_FILESYSTEM_HOST_SEED = b"NVX-HOST-TO-GUEST" PLATFORM_FILESYSTEM_GUEST_RESULT = b"NVX-GUEST-TO-HOST" +# A platform restore attaches the aggregate of the filesystem-shares scenario +# to the filesystem slot that the capture left dormant. The clone keeps its +# captured command line, which names no filesystem, so it mounts the aggregate +# itself and checks each child's kind, mode, and policy through its root. +PLATFORM_AGGREGATE_FAILURE_MARKER = b"NVX-PLATFORM-AGGREGATE-FAIL" +PLATFORM_AGGREGATE_MARKER = b"NVX-PLATFORM-AGGREGATE-OK" +# The restores of the platform snapshot that OpenVMM refuses before the guest +# runs, because their aggregates are ones that a cold boot refuses, and why. +PLATFORM_AGGREGATE_REJECTED = ( + ("overlapping-children", b"must not overlap"), + ("file-in-directory", b"must not overlap"), + ("directory-as-file", b"not a regular file"), + ("file-as-directory", b"not a plain directory"), + ("repeated-name", b"not unique"), + ("file-policy", b"exposes the file"), + ("outside-policy", b"outside the filesystem export roots"), + ("relative-policy", b"absolute host path"), +) +# A Windows host refuses caller ownership at a restore as at a cold boot. +PLATFORM_AGGREGATE_OWNER_REJECTION = b"--mount-owner caller requires a Linux host" +# Lines that only a restored clone's guest prints, from its platform repair's +# report to the restore's end, so that a refused restore prints none of them. +PLATFORM_AGGREGATE_GUEST_MARKERS = ( + PLATFORM_POST_RESTORE_MARKER, + PLATFORM_AGGREGATE_MARKER, + RESTORE_MARKER, +) +# The dormant slot of a snapshot of any other tier, or without a tier, binds +# only a --mount at a restore. +DORMANT_AGGREGATE_REJECTION = ( + b"--mount-aggregate on a dormant filesystem slot requires a platform snapshot" +) # Records the time ABI rates and restore clock report of each restore. These # records are written before the restored VPs run, so they never split a guest # console line; targets that log while the guest runs, such as virt_kvm's @@ -5954,6 +5988,209 @@ def run_platform_filesystem( raise RuntimeError(f"{context} modified snapshot artifacts") +def run_platform_aggregate_export( + executable: Path, + kernel: Path, + initrd: Path, + backend: str, + *, + memory_mib: int, + timeout: float, + output_dir: Path, +) -> None: + """Attach an aggregate of host directories and files to the filesystem + slot that a platform snapshot left dormant, and use it from the clone. + + The aggregate is the filesystem-shares scenario's: a read-write workspace + and a read-only tool cache, each with a denied path, a read-write data + directory whose root exposes only its allowed paths and whose notes alone + are writable, and a read-only and a read-write file. The clone keeps its + captured command line, so its guest mounts the aggregate itself, checks + each child's kind, mode, and policy through the aggregate's root, and binds + a directory and a file child; the host then checks that the guest changed + exactly what the policy lets it change. Before the guest runs, OpenVMM must + refuse the restores whose aggregates a cold boot refuses, and on a Windows + host caller ownership, and no restore may change the snapshot. + """ + with tempfile.TemporaryDirectory( + prefix="nvx-platform-aggregate-export-" + ) as temporary: + root = Path(temporary) + snapshot_path = root / "snapshot" + layer = root / "distro.erofs" + capture_scratch = root / "capture-scratch.raw" + restore_scratch = root / "restore-scratch.raw" + outside = root / "outside" + _write_pattern(layer, 1024 * 1024, 0x3C) + _write_pattern(capture_scratch, 1024 * 1024, 0xA5) + _write_pattern(restore_scratch, 1024 * 1024, 0x5A) + outside.mkdir() + workspace, toolcache, data, files = _create_shared_trees(root) + settings, output = _shared_files(files) + toolcache_contents = _tree_contents(toolcache) + data_contents = _tree_contents(data) + files_contents = _tree_contents(files) + boot_command = workload_boot_command( + executable, backend, kernel, initrd, memory_mib, "" + ) + # Without --mount or --mount-aggregate, the capture leaves the + # filesystem slot dormant. + capture_snapshot( + [ + *boot_command, + "--snapshot-destination", + str(snapshot_path), + "--snapshot-tier", + "platform", + "--microvm-sandbox-block", + _block_arg("distro", layer, read_only=True), + "--microvm-sandbox-block", + _block_arg("scratch", capture_scratch, read_only=False), + ], + snapshot_path, + timeout=timeout, + processors=1, + post_restore_script=_read_script("platform-aggregate-export.sh"), + log_path=output_dir / "platform-aggregate-export-capture.log", + ) + fingerprint = _scratch_snapshot_fingerprint(snapshot_path) + + def restore_command(*arguments: str) -> list[str]: + command = snapshot_restore_command(executable, backend, snapshot_path) + command.extend( + ( + "--microvm-sandbox-block", + _block_arg("distro", layer, read_only=True), + "--microvm-sandbox-block", + _block_arg("scratch", restore_scratch, read_only=False), + *arguments, + ) + ) + return command + + aggregate = ( + *_aggregate_arguments(workspace, toolcache, data, files), + "--mount-deny", + str(workspace / "secrets"), + "--mount-deny", + str(toolcache / "credentials"), + ) + context = "platform aggregate export restore" + restored = _measure_restore( + restore_command(*aggregate), + context=context, + environment=_restore_environment(), + timeout=timeout, + log_path=output_dir / "platform-aggregate-export-restore.log", + failure_marker=PLATFORM_AGGREGATE_FAILURE_MARKER, + ) + for marker in (PLATFORM_POST_RESTORE_MARKER, PLATFORM_AGGREGATE_MARKER): + if not contains_output_line(restored, marker): + raise RuntimeError(f"{context} did not report {marker.decode()!r}") + guest_write = workspace / "from-guest" + if ( + not guest_write.is_file() + or guest_write.read_bytes() != b"NVX-GUEST-WRITE\n" + ): + raise RuntimeError(f"{context} did not write to the read-write directory") + if not (workspace / "guest-directory").is_dir(): + raise RuntimeError(f"{context} did not create the guest's directory") + if (workspace / "secrets" / "token").read_bytes() != b"NVX-SECRET": + raise RuntimeError(f"{context} changed a denied path") + if _tree_contents(toolcache) != toolcache_contents: + raise RuntimeError(f"{context} changed the read-only directory") + data_contents["notes one.txt"] = b"NVX-NOTES\nNVX-GUEST-NOTE\n" + if _tree_contents(data) != data_contents: + raise RuntimeError( + f"{context} changed the data directory beyond its writable path" + ) + files_contents["results/output.txt"] = b"NVX-GUEST-OUTPUT\n" + if _tree_contents(files) != files_contents: + raise RuntimeError( + f"{context} changed the shared files' directories, the read-only " + "file, or the read-write file other than as it wrote" + ) + if _scratch_snapshot_fingerprint(snapshot_path) != fingerprint: + raise RuntimeError(f"{context} modified snapshot artifacts") + workspace_contents = _tree_contents(workspace) + + rejected_arguments = { + "overlapping-children": ( + "--mount-aggregate", + "/run/nvx/shares", + "--mount-child", + f"0,{toolcache},ro", + "--mount-child", + f"1,{toolcache / 'tools'},ro", + ), + "file-in-directory": ( + "--mount-aggregate", + "/run/nvx/shares", + "--mount-child", + f"0,{settings.parent},rw", + "--mount-child", + f"1,{settings},ro,file", + ), + "directory-as-file": ( + "--mount-aggregate", + "/run/nvx/shares", + "--mount-child", + f"0,{settings.parent},ro,file", + ), + "file-as-directory": ( + "--mount-aggregate", + "/run/nvx/shares", + "--mount-child", + f"0,{settings},ro", + ), + "repeated-name": ( + "--mount-aggregate", + "/run/nvx/shares", + "--mount-child", + f"same,{workspace},rw", + "--mount-child", + f"same,{toolcache},ro", + ), + "file-policy": (*aggregate, "--mount-deny", str(output)), + "outside-policy": (*aggregate, "--mount-deny", str(outside)), + "relative-policy": (*aggregate, "--mount-deny", "secrets"), + } + rejected = [ + (name, rejected_arguments[name], reason) + for name, reason in PLATFORM_AGGREGATE_REJECTED + ] + if sys.platform == "win32": + rejected.append( + ( + "caller-owner", + (*aggregate, "--mount-owner", "caller"), + PLATFORM_AGGREGATE_OWNER_REJECTION, + ) + ) + for name, arguments, reason in rejected: + _expect_process_failure( + restore_command(*arguments), + output_dir / f"platform-aggregate-export-rejected-{name}.log", + timeout, + reason, + forbidden_markers=PLATFORM_AGGREGATE_GUEST_MARKERS, + ) + if _scratch_snapshot_fingerprint(snapshot_path) != fingerprint: + raise RuntimeError( + "a refused platform aggregate export restore modified snapshot " + "artifacts" + ) + if ( + _tree_contents(workspace) != workspace_contents + or _tree_contents(toolcache) != toolcache_contents + or _tree_contents(data) != data_contents + or _tree_contents(files) != files_contents + ): + raise RuntimeError( + "a refused platform aggregate export restore changed a shared tree" + ) + + def run_restore_downtime( executable: Path, kernel: Path, @@ -6902,6 +7139,19 @@ def run_filesystem_snapshot( (late_root / "guest-result").read_bytes() != b"NVX-GUEST-TO-HOST" ): raise RuntimeError("dormant filesystem late attachment failed") + # Only a platform snapshot's dormant slot binds an aggregate. + aggregate_restore = snapshot_restore_command( + executable, backend, dormant_snapshot + ) + aggregate_restore.extend( + ("--mount-aggregate", "/mnt/late", "--mount-child", f"0,{late_root},rw") + ) + _expect_process_failure( + aggregate_restore, + output_dir / "filesystem-dormant-aggregate.log", + timeout, + DORMANT_AGGREGATE_REJECTION, + ) if _snapshot_fingerprint(dormant_snapshot) != dormant_fingerprint: raise RuntimeError( "dormant filesystem restores modified snapshot artifacts" @@ -7152,6 +7402,38 @@ def _shares_command( return command +def _create_shared_trees(root: Path) -> tuple[Path, Path, Path, Path]: + """Create the workspace, the tool cache, the data directory, and the + directory of the shared files that the aggregate of + `_aggregate_arguments` shares, below `root`, and return them. Each holds + paths that the aggregate's policy hides or exposes, and each shared file a + sibling that the aggregate does not share.""" + workspace = root / "workspace" + toolcache = root / "toolcache" + data = root / "data" + files = root / "files" + settings, output = _shared_files(files) + (workspace / "secrets").mkdir(parents=True) + (toolcache / "tools").mkdir(parents=True) + (toolcache / "credentials").mkdir() + (data / "public").mkdir(parents=True) + settings.parent.mkdir(parents=True) + output.parent.mkdir(parents=True) + (workspace / "seed").write_bytes(b"NVX-WORKSPACE") + (workspace / "secrets" / "token").write_bytes(b"NVX-SECRET") + (toolcache / "seed").write_bytes(b"NVX-TOOLCACHE") + (toolcache / "tools" / "node").write_bytes(b"NVX-TOOL") + (toolcache / "credentials" / "token").write_bytes(b"NVX-CREDENTIAL") + (data / "public" / "readme").write_bytes(b"NVX-PUBLIC") + (data / "notes one.txt").write_bytes(b"NVX-NOTES\n") + (data / "private.txt").write_bytes(b"NVX-PRIVATE") + settings.write_bytes(b"NVX-SETTINGS") + (settings.parent / "secret.json").write_bytes(b"NVX-CONFIG-SECRET") + output.write_bytes(b"NVX-OUTPUT\n") + (output.parent / "other.txt").write_bytes(b"NVX-OTHER") + return workspace, toolcache, data, files + + def run_filesystem_shares( executable: Path, kernel: Path, @@ -7167,29 +7449,8 @@ def run_filesystem_shares( its directory, together, as the children of one aggregate.""" with tempfile.TemporaryDirectory(prefix="nvx-filesystem-shares-") as temporary: root = Path(temporary) - workspace = root / "workspace" - toolcache = root / "toolcache" - data = root / "data" - files = root / "files" + workspace, toolcache, data, files = _create_shared_trees(root) settings, output = _shared_files(files) - (workspace / "secrets").mkdir(parents=True) - (toolcache / "tools").mkdir(parents=True) - (toolcache / "credentials").mkdir() - (data / "public").mkdir(parents=True) - settings.parent.mkdir(parents=True) - output.parent.mkdir(parents=True) - (workspace / "seed").write_bytes(b"NVX-WORKSPACE") - (workspace / "secrets" / "token").write_bytes(b"NVX-SECRET") - (toolcache / "seed").write_bytes(b"NVX-TOOLCACHE") - (toolcache / "tools" / "node").write_bytes(b"NVX-TOOL") - (toolcache / "credentials" / "token").write_bytes(b"NVX-CREDENTIAL") - (data / "public" / "readme").write_bytes(b"NVX-PUBLIC") - (data / "notes one.txt").write_bytes(b"NVX-NOTES\n") - (data / "private.txt").write_bytes(b"NVX-PRIVATE") - settings.write_bytes(b"NVX-SETTINGS") - (settings.parent / "secret.json").write_bytes(b"NVX-CONFIG-SECRET") - output.write_bytes(b"NVX-OUTPUT\n") - (output.parent / "other.txt").write_bytes(b"NVX-OTHER") toolcache_contents = _tree_contents(toolcache) files_contents = _tree_contents(files) @@ -9101,6 +9362,21 @@ def _run_scenarios( timeout=args.timeout, output_dir=output_dir, ) + if "platform-aggregate-export" in scenarios: + report.start( + "platform-aggregate-export", + "Running microVM platform-snapshot aggregate export correctness on " + f"OpenVMM/{args.backend}", + ) + run_platform_aggregate_export( + executable, + kernel, + initrd, + args.backend, + memory_mib=args.memory_mib, + timeout=args.timeout, + output_dir=output_dir, + ) if "restore-downtime" in scenarios: report.start( "restore-downtime", diff --git a/scripts/test_microvm_tests.py b/scripts/test_microvm_tests.py index 403a6366a..3a40ce939 100644 --- a/scripts/test_microvm_tests.py +++ b/scripts/test_microvm_tests.py @@ -11086,6 +11086,528 @@ def test_platform_filesystem_keeps_the_snapshot_intact(self): fingerprints=[original, changed], ) + def test_platform_aggregate_export_script_mounts_and_checks_the_aggregate(self): + script = microvm_tests._read_script("platform-aggregate-export.sh") + # The clone keeps its captured command line, which discovers the slot + # but names no filesystem, so the guest mounts the aggregate itself. + self.assertIn( + "\ngrep -q 'virtio_mmio.device=0x1000@0xd0001000:6' /proc/cmdline ||", + script, + ) + self.assertIn( + "\nif grep -Eq '(^| )(virtfs_|nvx_share=)' /proc/cmdline;", script + ) + self.assertIn('\nmount -t virtiofs microvm "$shares" ||', script) + self.assertIn(r"""[ "$(ls "$shares" | tr '\n' ' ')" = "0 1 2 3 4 " ]""", script) + # It checks every child of the aggregate, and binds a directory and a + # file child where it needs them. + for child in range(5): + self.assertIn(f'"$shares/{child}', script) + self.assertIn('\nmount --bind "$shares/0" /workspace ||', script) + self.assertIn('\nmount --bind "$shares/3" /config/settings.json ||', script) + self.assertIn( + "\n echo " + f'"{microvm_tests.PLATFORM_AGGREGATE_FAILURE_MARKER.decode()} $1"\n', + script, + ) + self.assertTrue( + script.endswith( + f"\necho {microvm_tests.PLATFORM_AGGREGATE_MARKER.decode()}\n" + ) + ) + + def _run_platform_aggregate_export( + self, + output_dir: Path, + measure_once: MagicMock, + *, + fingerprints: Sequence[tuple[str, str, str, str | None]] | None = None, + refused: Callable[..., None] | None = None, + platform: str = "linux", + ) -> tuple[MagicMock, MagicMock, MagicMock]: + fingerprint = ("manifest", "state", "memory", None) + with ( + patch.object( + microvm_tests, + "workload_boot_command", + return_value=["openvmm", "boot"], + ) as workload_boot_command, + patch.object(microvm_tests, "capture_snapshot") as capture_snapshot, + patch.object(microvm_tests, "measure_once", measure_once), + patch.object( + microvm_tests, + "_scratch_snapshot_fingerprint", + side_effect=fingerprints or itertools.repeat(fingerprint), + ), + patch.object( + microvm_tests, "_expect_process_failure", side_effect=refused + ) as expect_failure, + patch.object(microvm_tests.sys, "platform", platform), + ): + microvm_tests.run_platform_aggregate_export( + Path("openvmm"), + Path("vmlinux"), + Path("initrd"), + "kvm", + memory_mib=128, + timeout=60, + output_dir=output_dir, + ) + return workload_boot_command, capture_snapshot, expect_failure + + @staticmethod + def _aggregate_children(command: list[str]) -> list[tuple[str, Path, str]]: + """Return the name, host path, and flags of each `--mount-child` of + `command`, in order.""" + children: list[tuple[str, Path, str]] = [] + for index, argument in enumerate(command): + if argument == "--mount-child": + name, rest = command[index + 1].split(",", 1) + host_path, flags = rest.split(",", 1) if "," in rest else (rest, "") + children.append((name, Path(host_path), flags)) + return children + + @staticmethod + def _option_values(command: list[str], option: str) -> list[str]: + return [ + command[index + 1] + for index, argument in enumerate(command) + if argument == option + ] + + @staticmethod + def _aggregate_guest(children: list[tuple[str, Path, str]]) -> None: + """Change the host trees as the clone's guest does through the + aggregate.""" + roots = {name: host_path for name, host_path, _ in children} + (roots["0"] / "from-guest").write_bytes(b"NVX-GUEST-WRITE\n") + (roots["0"] / "guest-directory").mkdir() + with (roots["2"] / "notes one.txt").open("ab") as notes: + notes.write(b"NVX-GUEST-NOTE\n") + roots["4"].write_bytes(b"NVX-GUEST-OUTPUT\n") + + @classmethod + def _platform_aggregate_measure( + cls, + output: bytes | None = None, + *, + guest: Callable[[list[tuple[str, Path, str]]], None] | None = None, + on_restore: Callable[[list[str]], None] | None = None, + ) -> MagicMock: + def measure(command: list[str], **kwargs: object) -> None: + if on_restore is not None: + on_restore(command) + (guest or cls._aggregate_guest)(cls._aggregate_children(command)) + log_path = cast(Path, kwargs["log_path"]) + log_path.parent.mkdir(parents=True, exist_ok=True) + log_path.write_bytes( + output + if output is not None + else microvm_tests.PLATFORM_POST_RESTORE_MARKER + + b"\r\n" + + microvm_tests.PLATFORM_AGGREGATE_MARKER + + b"\r\n" + + benchmark.RESTORE_MARKER + + b"\r\n" + ) + + return MagicMock(side_effect=measure) + + def test_platform_aggregate_export_attaches_an_aggregate_at_the_restore(self): + def blocks(command: list[str]) -> list[str]: + return self._option_values(command, "--microvm-sandbox-block") + + trees: dict[str, dict[str, bytes | None]] = {} + + def on_restore(command: list[str]) -> None: + for name, host_path, flags in self._aggregate_children(command): + trees[name] = ( + {"": host_path.read_bytes()} + if flags.endswith("file") + else microvm_tests._tree_contents(host_path) + ) + + # Each refused restore requests an aggregate that a cold boot refuses + # for the reason that the scenario expects. + refusals: list[str] = [] + + def refused(command: list[str], *args: object, **_kwargs: object) -> None: + reason = cast(bytes, args[2]) + children = self._aggregate_children(command) + roots = [host_path for _, host_path, _ in children] + denied = [ + Path(path) for path in self._option_values(command, "--mount-deny") + ] + if reason == b"must not overlap": + (_, first, _), (_, second, _) = children + self.assertTrue(second.is_relative_to(first)) + refusals.append( + "overlap" if first.is_dir() and second.is_dir() else "file" + ) + elif reason == b"not a regular file": + ((_, host_path, flags),) = children + self.assertTrue(host_path.is_dir()) + self.assertEqual(flags, "ro,file") + refusals.append("directory-as-file") + elif reason == b"not a plain directory": + ((_, host_path, flags),) = children + self.assertTrue(host_path.is_file()) + self.assertEqual(flags, "ro") + refusals.append("file-as-directory") + elif reason == b"not unique": + self.assertEqual([name for name, _, _ in children], ["same", "same"]) + self.assertTrue(all(root.is_dir() for root in roots)) + refusals.append("repeated-name") + elif reason == b"exposes the file": + file_children = [ + host_path + for _, host_path, flags in children + if flags.endswith("file") + ] + self.assertIn(denied[-1], file_children) + refusals.append("file-policy") + elif reason == b"outside the filesystem export roots": + self.assertTrue(denied[-1].is_dir()) + self.assertFalse(any(denied[-1].is_relative_to(root) for root in roots)) + refusals.append("outside-policy") + elif reason == b"absolute host path": + self.assertFalse(denied[-1].is_absolute()) + refusals.append("relative-policy") + else: + self.fail(f"unexpected refusal {reason!r}") + + with tempfile.TemporaryDirectory() as temporary: + output_dir = Path(temporary) / "logs" + measure_once = self._platform_aggregate_measure(on_restore=on_restore) + ( + workload_boot_command, + capture_snapshot, + expect_failure, + ) = self._run_platform_aggregate_export( + output_dir, measure_once, refused=refused + ) + logs = sorted(path.name for path in output_dir.iterdir()) + + self.assertIn("platform-aggregate-export", microvm_tests.MICROVM_TEST_SCENARIOS) + self.assertIn( + "platform-aggregate-export", microvm_tests.SANDBOX_CONTROL_SCENARIOS + ) + # A platform command line carries no caller tokens, and the capture + # leaves the filesystem slot dormant. + self.assertEqual(workload_boot_command.call_args.args[5], "") + capture_command = capture_snapshot.call_args.args[0] + self.assertNotIn("--mount", capture_command) + self.assertNotIn("--mount-aggregate", capture_command) + self.assertEqual( + capture_command[capture_command.index("--snapshot-tier") + 1], "platform" + ) + capture_blocks = blocks(capture_command) + self.assertEqual(len(capture_blocks), 2) + self.assertRegex(capture_blocks[0], r"^distro:file:.+,ro$") + self.assertRegex(capture_blocks[1], r"^scratch:file:.+(? Callable[[list[tuple[str, Path, str]]], None]: + def guest(children: list[tuple[str, Path, str]]) -> None: + self._aggregate_guest(children) + change({name: host_path for name, host_path, _ in children}) + + return guest + + def skip_write(children: list[tuple[str, Path, str]]) -> None: + roots = {name: host_path for name, host_path, _ in children} + (roots["0"] / "guest-directory").mkdir() + + def skip_directory(children: list[tuple[str, Path, str]]) -> None: + roots = {name: host_path for name, host_path, _ in children} + (roots["0"] / "from-guest").write_bytes(b"NVX-GUEST-WRITE\n") + + for name, guest, message in ( + ("missing write", skip_write, "did not write to the read-write directory"), + ( + "missing directory", + skip_directory, + "did not create the guest's directory", + ), + ( + "denied path", + guest_then( + lambda roots: (roots["0"] / "secrets" / "token").write_bytes(b"x") + ), + "changed a denied path", + ), + ( + "read-only directory", + guest_then(lambda roots: (roots["1"] / "mutation").write_bytes(b"x")), + "changed the read-only directory", + ), + ( + "data beyond the notes", + guest_then( + lambda roots: (roots["2"] / "private.txt").write_bytes(b"x") + ), + "changed the data directory beyond its writable path", + ), + ( + "read-only file", + guest_then(lambda roots: roots["3"].write_bytes(b"x")), + "changed the shared files' directories", + ), + ( + "file's directory", + guest_then( + lambda roots: (roots["4"].parent / "added").write_bytes(b"x") + ), + "changed the shared files' directories", + ), + ): + with ( + self.subTest(name=name), + tempfile.TemporaryDirectory() as temporary, + ): + with self.assertRaisesRegex( + RuntimeError, "^platform aggregate export restore " + message + ): + self._run_platform_aggregate_export( + Path(temporary), self._platform_aggregate_measure(guest=guest) + ) + + def test_platform_aggregate_export_reports_guest_failures(self): + failure = benchmark.GuestFailureReported( + "NVX-PLATFORM-AGGREGATE-FAIL mount", "tail" + ) + with tempfile.TemporaryDirectory() as temporary: + with self.assertRaises(RuntimeError) as raised: + self._run_platform_aggregate_export( + Path(temporary), MagicMock(side_effect=failure) + ) + self.assertEqual( + str(raised.exception), + "platform aggregate export restore: guest reported " + "NVX-PLATFORM-AGGREGATE-FAIL mount\n" + "--- OpenVMM output ---\ntail", + ) + + def test_platform_aggregate_export_keeps_the_snapshot_and_the_trees_intact(self): + original = ("manifest", "state", "memory", None) + changed = ("manifest", "state", "changed", None) + for fingerprints, message in ( + # The capture, then the clone. + ( + [original, changed], + "^platform aggregate export restore modified snapshot artifacts$", + ), + # The capture, the clone, then the refused restores. + ( + [original, original, changed], + "^a refused platform aggregate export restore modified snapshot " + "artifacts$", + ), + ): + with ( + self.subTest(message=message), + tempfile.TemporaryDirectory() as temporary, + ): + with self.assertRaisesRegex(RuntimeError, message): + self._run_platform_aggregate_export( + Path(temporary), + self._platform_aggregate_measure(), + fingerprints=fingerprints, + ) + + def refused_then( + change: Callable[[dict[str, Path]], object], + ) -> Callable[..., None]: + def refused(command: list[str], *_args: object, **_kwargs: object) -> None: + # A refused restore of the whole aggregate that reached the + # guest might write to a share. + children = self._aggregate_children(command) + if len(children) == 5: + change({name: host_path for name, host_path, _ in children}) + + return refused + + changes: tuple[tuple[str, Callable[[dict[str, Path]], object]], ...] = ( + ( + "workspace", + lambda roots: (roots["0"] / "secrets" / "token").write_bytes(b"x"), + ), + ("data", lambda roots: (roots["2"] / "private.txt").write_bytes(b"x")), + ) + for tree, change in changes: + with ( + self.subTest(tree=tree), + tempfile.TemporaryDirectory() as temporary, + ): + with self.assertRaisesRegex( + RuntimeError, + "^a refused platform aggregate export restore changed a shared " + "tree$", + ): + self._run_platform_aggregate_export( + Path(temporary), + self._platform_aggregate_measure(), + refused=refused_then(change), + ) + def test_restore_downtime_shares_one_window_and_waits_for_the_restore_marker( self, ):