vmm_tests: add virtio-villain conformance test runner

[jstarks]

virtio-villain is a guest-side virtio fault-injection and conformance suite. Its static musl init drives the virtio transports itself and feeds deliberately out-of-spec virtqueue inputs to the device model, printing a pass/fail verdict per test on the serial console. This PR wires it into OpenVMM as an automated test suite so that malformed-descriptor handling and other virtio robustness bugs are caught by CI rather than in production.

The new virtio_villain_tests crate uses petri as a library together with libtest-mimic to expose one nextest case per villain test. Each case boots a single kitchen-sink OpenVMM VM -- every virtio device villain can probe, attached over PCIe or, for the MMIO-transport tests, the virtio-mmio bus -- with vv.test=<id> on the kernel command line, waits for the guest to halt, and reads the verdict from the teed serial log. Villain's initramfs and tests.tsv are supplied via VILLAIN_INITRAMFS/VILLAIN_TSV (resolved from an openvmm-deps release in CI), and the guest kernel is reused from the existing linux-direct test artifact.

Not every villain test can run against OpenVMM's kitchen-sink VM, so the runner classifies tests up front from the tsv metadata: cases targeting a device we don't attach, or requiring a feature bit or virtqueue count we don't offer, self-SKIP and are marked ignored instead of reported as false passes. The device capability table is derived directly from the virtio_spec feature and device-type definitions so it stays honest against the models. Separately, some malformed-descriptor tests drive OpenVMM's virtio worker into a non-terminating loop the VM cannot recover from or tear down; rather than paper over that product bug, per-test slow-timeout overrides in nextest terminate the run and report it as a failure, and the specific tests known to trigger it are marked ignored -- skipped in CI but still runnable locally while the bug is fixed.

Results are made visible on the logview website: the runner resolves a per-test log directory through petri's TEST_LOG_DIRECTORY artifact, writes the per-test petri.jsonl plus the petri.passed/petri.failed markers the uploader counts, and initializes tracing per test. The petri initrd-override seam is generalized so a caller can supply its own fully-formed initrd.

On the pipeline side, villain runs as a separate parallel per-PR job on the x64 Linux KVM pool, split into a build-archive step and a consume-and-test step that mirror the existing vmm-tests flow and take the target triple like every other node. The build machine publishes the villain nextest archive alongside the OpenVMM binary; the test job consumes it, unpacks the villain guest artifact from openvmm-deps (bumped to 0.3.0-112, which now carries it), builds OpenVMM, stages the guest kernel, and runs the suite. The same wiring is exposed locally through xflowey vmm-tests-run.

A dev-guide page documents the suite, how to run it locally, and how to triage known failures.

[Copilot]

Pull request overview

Adds a new “virtio-villain” (guest-side virtio fault-injection / conformance) test suite runner that boots OpenVMM under petri, then wires it into Flowey pipelines and PR CI so results (JUnit + per-test petri logs) can be collected and published.

Changes:

• Introduces virtio_villain_tests crate: parses tests.tsv, runs one VM per villain test via petri, and surfaces results via libtest-mimic.
• Extends petri logging to support creating additional per-test log roots after global tracing is initialized.
• Adds Flowey nodes/jobs/pipelines + CI wiring to build a nextest archive and run it on KVM runners, publishing logs for logview.

Reviewed changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
vmm_tests/virtio_villain_tests/tests/villain.rs	Standalone libtest-mimic harness that enumerates villain tests and runs each in its own VM with per-test logging.
vmm_tests/virtio_villain_tests/src/villain.rs	TSV parsing + serial marker verdict scanning, with unit tests.
vmm_tests/virtio_villain_tests/src/run.rs	VM bring-up and “kitchen-sink” virtio device attachment, then serial log scanning.
vmm_tests/virtio_villain_tests/src/lib.rs	Public module surface for the new test runner crate.
vmm_tests/virtio_villain_tests/src/known_failures.rs	Central list of known-failing villain tests that are marked ignored.
vmm_tests/virtio_villain_tests/Cargo.toml	Defines the crate and its custom harness = false libtest-mimic test target.
petri/src/vm/openvmm/construct.rs	Allows a provided prebuilt_initrd to override firmware-provided initrd (needed for villain PID1 initramfs).
petri/src/tracing.rs	Adds new_log_source() to create additional log roots without reinstalling the global subscriber.
flowey/flowey_lib_hvlite/src/resolve_virtio_villain.rs	Flowey node to resolve virtio-villain guest artifacts (local override or GitHub release).
flowey/flowey_lib_hvlite/src/lib.rs	Exposes new Flowey nodes for virtio-villain build/resolve.
flowey/flowey_lib_hvlite/src/build_nextest_virtio_villain_tests.rs	Builds and archives the virtio-villain nextest suite for toolchain-free execution.
flowey/flowey_lib_hvlite/src/_jobs/run_virtio_villain_tests.rs	Local Flowey job to build OpenVMM, resolve artifacts, and run the suite end-to-end.
flowey/flowey_lib_hvlite/src/_jobs/mod.rs	Registers new virtio-villain job modules.
flowey/flowey_lib_hvlite/src/_jobs/consume_and_test_nextest_virtio_villain_archive.rs	CI job to unpack archive and run tests on KVM test machines, publishing JUnit + logs.
flowey/flowey_lib_hvlite/src/_jobs/cfg_versions.rs	Bumps OPENVMM_DEPS and wires resolve_virtio_villain config to that version.
flowey/flowey_hvlite/src/pipelines/virtio_villain_run.rs	Adds cargo xflowey virtio-villain-run pipeline entrypoint.
flowey/flowey_hvlite/src/pipelines/mod.rs	Registers the new pipeline subcommand.
flowey/flowey_hvlite/src/pipelines/checkin_gates.rs	Integrates build+consume split into PR gates, adding a dedicated virtio-villain CI job.
ci-flowey/openvmm-pr.yaml	Flowey-generated ADO pipeline wiring for the new archive and test job.
Cargo.toml	Adds vmm_tests/virtio_villain_tests to workspace members.
Cargo.lock	Adds lockfile entries for the new crate and deps.
.github/workflows/openvmm-pr.yaml	Flowey-generated GitHub workflow updates to run the new job and publish artifacts.
.github/workflows/openvmm-pr-release.yaml	Flowey-generated release PR workflow updates to include the new job.
.github/workflows/openvmm-ci.yaml	Flowey-generated CI workflow updates to include the new job and artifact wiring.
.config/nextest.toml	Adds per-package overrides (slow-timeout + threads-required) for virtio-villain tests.

Comments suppressed due to low confidence (4)

vmm_tests/virtio_villain_tests/src/known_failures.rs:55

• Known-failure reasons currently include a placeholder (microsoft/openvmm#TODO). This reads like a real issue link but isn't actionable; it’s better to either link a real tracking issue or make the TODO explicit without looking like a PR/issue reference.

    KnownFailure {
        name: "P0003",
        reason: "virtio-blk packed descriptor list exceeding queue size wedges \
                 the device (microsoft/openvmm#TODO)",
    },

vmm_tests/virtio_villain_tests/src/known_failures.rs:60

• Known-failure reason includes placeholder (microsoft/openvmm#TODO). Replace with a real issue link or an explicit TODO that doesn't look like an issue reference.

    KnownFailure {
        name: "T0001",
        reason: "virtio-blk self-looping descriptor chain hangs the virtio \
                 worker unrecoverably (microsoft/openvmm#TODO)",
    },

vmm_tests/virtio_villain_tests/src/known_failures.rs:65

• Known-failure reason includes placeholder (microsoft/openvmm#TODO). Replace with a real issue link or an explicit TODO that doesn't look like an issue reference.

    KnownFailure {
        name: "T0002",
        reason: "virtio-blk descriptor chain exceeding queue size hangs the \
                 virtio worker unrecoverably (microsoft/openvmm#TODO)",
    },

vmm_tests/virtio_villain_tests/src/known_failures.rs:70

• Known-failure reason includes placeholder (microsoft/openvmm#TODO). Replace with a real issue link or an explicit TODO that doesn't look like an issue reference.

    KnownFailure {
        name: "T0003",
        reason: "virtio-blk out-of-bounds descriptor `next` index hangs the \
                 virtio worker unrecoverably (microsoft/openvmm#TODO)",
    },

[Copilot]

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (2)

vmm_tests/virtio_villain_tests/src/known_failures.rs:49

• The reason strings in KNOWN_FAILURES still contain placeholder issue links (microsoft/openvmm#TODO). Since ignored tests are intended to track product bugs, it’s important that each entry links to a real issue so the skip list doesn’t become stale.

pub const KNOWN_FAILURES: &[KnownFailure] = &[
    KnownFailure {
        name: "B0002",
        reason: "virtio-blk sector*512+data_len 64-bit wrap wedges the device \
                 (microsoft/openvmm#TODO)",

vmm_tests/virtio_villain_tests/src/villain.rs:50

• parse_tsv silently maps a missing/malformed device_id column to 0 (unwrap_or("0x0000") + unwrap_or(0)). That can hide issues with a corrupted/incorrect tests.tsv and make debugging harder. Since this is test metadata, failing fast with a line-specific error would be more actionable.

        let device_id = cols.next().unwrap_or("0x0000");
        let device_id = device_id
            .strip_prefix("0x")
            .and_then(|h| u16::from_str_radix(h, 16).ok())
            .unwrap_or(0);

[github-actions]

23 Petri tests failed (0 unstable)

[Copilot]

Pull request overview

Copilot reviewed 28 out of 29 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (2)

petri/src/vm/openvmm/construct.rs:195

• If prebuilt_initrd is set but load_firmware() returns a non-Linux LoadMode (e.g. UEFI), the initrd override is silently ignored. That can lead to confusing behavior where callers believe they supplied an initrd but the VM boots with the firmware-provided one. It would be safer to error out when prebuilt_initrd is present but the load mode can’t accept an initrd.

        if let Some(prebuilt) = properties.prebuilt_initrd.as_ref() {
            if let LoadMode::Linux { initrd, .. } = &mut load_mode {
                let file = std::fs::File::open(prebuilt).with_context(|| {
                    format!("failed to open prebuilt initrd at {}", prebuilt.display())
                })?;
                *initrd = Some(file);
            }
        }

vmm_tests/virtio_villain_tests/src/known_failures.rs:52

• The known-failure reasons are currently placeholders ending in (microsoft/openvmm#TODO). If this list is intended to live long-term (and is used to suppress failures in CI), each entry should link to a real tracking issue so the ignore can be audited and retired. Please replace the TODOs with actual issue numbers (or drop the entry until an issue is filed).

/// Seeded from the first full-suite CI run (openvmm-deps 0.3.0-112, x86_64/KVM):
/// the tests below either drove OpenVMM's virtio worker into a non-terminating
/// loop (marked "host hang" — these time out, which is why the list is ignored
/// rather than inverted) or the device model accepted malformed input / returned
/// a wrong value. Issues still need to be filed; update each `reason` with the
/// issue link once they are.
pub const KNOWN_FAILURES: &[KnownFailure] = &[
    KnownFailure {
        name: "B0002",
        reason: "virtio-blk sector*512+data_len 64-bit wrap wedges the device \
                 (microsoft/openvmm#TODO)",
    },

[github-actions]

280 Petri tests failed (3 unstable)

[Copilot]

Pull request overview

Copilot reviewed 31 out of 32 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

vmm_tests/virtio_villain_tests/src/villain.rs:24

• VillainTest::device_id is documented (and unit-tested below) as a virtio device type value like 0x0002 for block, but the rest of the harness (e.g. supported_devices::SUPPORTED_DEVICE_IDS) uses virtio-PCI device IDs in the 0x1040 + device_type namespace. If tests.tsv uses virtio-PCI IDs (as implied elsewhere), this doc comment is misleading; if it uses types, the skip/ignore logic will ignore the wrong tests. Align this documentation with the ID space the runner actually expects.
  vmm_tests/virtio_villain_tests/src/known_failures.rs:122
• The RNG0004 rationale ends with a pointer to ai repo knowledge/..., which isn’t available in this repository. Please remove that reference or replace it with an in-repo link so external contributors can audit the decision.

                 OpenVMM is intentionally left as-is. Represents the villain \
                 `*_huge_len_past_ram` (\"crosses end of RAM\") family. See ai \
                 repo knowledge/context/virtio-huge-len-descriptor-validation.md",

[github-actions]

248 Petri tests failed (1 unstable)

[Copilot]

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

vmm_tests/virtio_villain_tests/src/known_failures.rs:56

• The reason strings use (microsoft/openvmm#TODO) placeholders, which look like real issue references but aren’t valid links. Before merging, these should either be replaced with actual issue numbers or the placeholder link text should be removed/rewritten so it can’t be mistaken for a real tracker reference.

    KnownFailure {
        name: "B0002",
        reason: "virtio-blk sector*512+data_len 64-bit wrap wedges the device \
                 (microsoft/openvmm#TODO)",
    },

[github-actions]

213 Petri tests failed (1 unstable)

[Copilot]

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (3)

vmm_tests/virtio_villain_tests/src/villain.rs:162

• parse_tsv() claims strict parsing, but extra trailing columns after min_queues are currently ignored. If virtio-villain extends tests.tsv with new columns, this could silently mask a format change; consider rejecting any extra columns after the expected 8.
  petri/src/tracing.rs:302
• new_log_source() canonicalizes root_path, which fails if the directory doesn't already exist. Since this is a public helper intended for per-test log directories, it would be more ergonomic/robust to create the directory before canonicalizing (or explicitly document the requirement).

pub fn new_log_source(root_path: &Path) -> anyhow::Result<PetriLogSource> {
    // Canonicalize so that printed attachment paths are most likely to work.
    let root_path = root_path.fs_err_canonicalize()?;
    let jsonl = File::create(root_path.join("petri.jsonl"))?;

Guide/src/dev_guide/tests/virtio_villain.md:80

• Follow-up sentence still refers to SUPPORTED_DEVICE_IDS; the code path to enable more devices is to add a new entry to DEVICE_CAPS and attach the device in attach_kitchen_sink().

yet emulate (IOMMU, memory balloon, virtio-mem, watchdog, RTC). To enable a
device's tests, add its ID to `SUPPORTED_DEVICE_IDS` and attach the device in
`attach_kitchen_sink` (`run.rs`); keep the two in sync.

[github-actions]

4 Petri tests failed (1 unstable)

[Copilot]

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (4)

vmm_tests/virtio_villain_tests/src/run.rs:289

• Same issue as the MMIO branch: silently not updating the kernel cmdline if load_mode isn't Linux can cause confusing failures. Prefer failing fast with an explicit message.
  Guide/src/dev_guide/tests/virtio_villain.md:61
• supported_devices.rs doesn't define SUPPORTED_DEVICE_IDS; it defines DEVICE_CAPS (device id + offered features + queue count). Referencing the correct symbol helps readers keep the guide in sync with the actual code.

Villain has roughly 1400 tests but only a handful of device classes. Rather than
list every test to skip, we list the device IDs the kitchen-sink VM attaches, in
`SUPPORTED_DEVICE_IDS` (IDs follow the virtio convention `0x1040 +
virtio_device_type`):

Guide/src/dev_guide/tests/virtio_villain.md:80

• The enablement instructions reference SUPPORTED_DEVICE_IDS, but the code uses DEVICE_CAPS as the source of truth (and the VM wiring in attach_kitchen_sink). Updating this makes the “keep in sync” guidance actionable.

yet emulate (IOMMU, memory balloon, virtio-mem, watchdog, RTC). To enable a
device's tests, add its ID to `SUPPORTED_DEVICE_IDS` and attach the device in
`attach_kitchen_sink` (`run.rs`); keep the two in sync.

Guide/src/dev_guide/tests/virtio_villain.md:76

• This says device-agnostic tests (device_id == 0x0000) always run, but supported_devices::expected_skip intentionally ignores PCI transport tests with device_id == 0 due to a villain harness limitation. The guide should reflect that so skipped transport tests aren’t surprising.

A test whose target device is not in this set is ignored when its trial is
constructed, so it never boots a VM. Device-agnostic tests (`device_id ==
0x0000`, such as transport-level PCI checks) always run.

[github-actions]

3 Petri tests failed (2 unstable)

[Copilot]

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (5)

Guide/src/dev_guide/tests/virtio_villain.md:113

• This example uses cargo xflowey virtio-villain-run, but the suite is selected via cargo xflowey vmm-tests-run --suite virtio-villain in this PR.

```bash
cargo xflowey virtio-villain-run --filter "test(B0001)"

**Guide/src/dev_guide/tests/virtio_villain.md:122**
* This example uses `cargo xflowey virtio-villain-run`, but the suite is selected via `cargo xflowey vmm-tests-run --suite virtio-villain` in this PR.

cargo xflowey virtio-villain-run --run-ignored --filter "test(B0002)"

**vmm_tests/virtio_villain_tests/tests/villain.rs:182**
* `petri::try_init_tracing` installs a process-global subscriber and can only succeed once per process. If this harness binary is invoked directly (rather than by nextest running a single trial per process), the second selected trial will fail with a confusing tracing-init error. Consider improving the error context so the failure clearly explains the required execution model (one trial per process via `cargo nextest`).
**Guide/src/dev_guide/tests/virtio_villain.md:36**
* The docs mention local overrides via `--villain-initramfs` / `--villain-tsv`, but this PR’s runner uses only the `VILLAIN_INITRAMFS` / `VILLAIN_TSV` environment variables (and CI injects those). As written, readers will look for CLI flags that don’t exist.

existing OpenVMM linux-direct test vmlinux. Villain's initramfs.cpio.gz and
tests.tsv come from the openvmm-deps release artifact, or locally from
--villain-initramfs / --villain-tsv (or the VILLAIN_INITRAMFS /
VILLAIN_TSV environment variables).

**Guide/src/dev_guide/tests/virtio_villain.md:106**
* The command `cargo xflowey virtio-villain-run` is not defined in this repo (the runner is wired into `cargo xflowey vmm-tests-run` via `--suite virtio-villain`). This section should point to the actual CLI so local runs are discoverable.

This issue also appears in the following locations of the same file:
- line 111
- line 120

cargo xflowey virtio-villain-run builds OpenVMM, downloads the villain artifact,
stages the guest kernel, and runs the suite:

cargo xflowey virtio-villain-run

</details>

[github-actions]

4 Petri tests failed (2 unstable)

[Copilot]

Pull request overview

Copilot reviewed 33 out of 34 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

Guide/src/dev_guide/tests/virtio_villain.md:106

• The documented flowey command cargo xflowey virtio-villain-run doesn't appear to exist in this PR (the new wiring is exposed via vmm-tests-run --suite virtio-villain). The examples in this section should use the actual CLI so local runs match the implementation.

`cargo xflowey virtio-villain-run` builds OpenVMM, downloads the villain artifact,
stages the guest kernel, and runs the suite:

```bash
cargo xflowey virtio-villain-run

**vmm_tests/virtio_villain_tests/src/supported_devices.rs:99**
* The `DEVICE_CAPS` comment points at `vm/devices/net/virtio_net/...`, but the virtio-net crate lives under `vm/devices/virtio/virtio_net/...` in this repo. Also, the parenthetical "ring features included" is misleading given some devices (e.g. vsock) do not advertise ring features in `traits()`. Updating the comment will make it easier to keep this table in sync with the actual models.
**Guide/src/dev_guide/tests/virtio_villain.md:36**
* The guide mentions `--villain-initramfs` / `--villain-tsv`, but there are no flowey/CLI options with those names in this PR (the runner only supports `VILLAIN_INITRAMFS` / `VILLAIN_TSV` env var overrides). This section should be updated to avoid sending readers to non-existent flags.

This issue also appears on line 101 of the same file.

existing OpenVMM linux-direct test vmlinux. Villain's initramfs.cpio.gz and
tests.tsv come from the openvmm-deps release artifact, or locally from
--villain-initramfs / --villain-tsv (or the VILLAIN_INITRAMFS /
VILLAIN_TSV environment variables).

**vmm_tests/virtio_villain_tests/src/supported_devices.rs:82**
* This doc comment is inaccurate: not every device model advertises these ring feature bits (e.g. `virtio_vsock`'s `traits()` does not set them). The comment should be softened to avoid implying `RING_FEATURES` is universally present.

This issue also appears on line 95 of the same file.
</details>

[github-actions]

3 Petri tests failed (1 unstable)

[smalis-msft]

Could/should the timeouts be lower?

[smalis-msft]

I feel like we have more things that could go here? Like maybe the incubator stuff?

[smalis-msft]

Do all the options here apply to all suites?

[smalis-msft]

I worry that we're gaining too many arguments in nodes like this, and that there could be combinations of arguments that look valid but break things.

[smalis-msft]

Or that just the cognitive burden of adding some new caller of this node and having to figure out all these options is too high.

[smalis-msft]

We really need to rewrite our cli so that as much of this validation as is reasonable is done automatically or by clap.