Skip to content

[DO NOT REVIEW] microVM: refuse virtio device buffers outside the e820 RAM ranges - #4794

Merged
Pedro Henrique Penna (ppenna) merged 5 commits into
feature/microvm/devfrom
copilot/microvm-virtio-ram-only-dma
Oct 29, 2026
Merged

[DO NOT REVIEW] microVM: refuse virtio device buffers outside the e820 RAM ranges#4794
Pedro Henrique Penna (ppenna) merged 5 commits into
feature/microvm/devfrom
copilot/microvm-virtio-ram-only-dma

Conversation

@ppenna

@ppenna Pedro Henrique Penna (ppenna) commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Restricts the DMA of every microVM virtio device to the RAM ranges of the e820 map, for microsoft/nvx#554. A device now refuses a queue whose rings, or a descriptor whose buffer, lies even partly in the shared virtio interrupt-status page, the ISA hole, or the MMIO aperture: it reads and writes none of it, and its transport marks it as needing reset.

Based on feature/microvm/dev. This branch is five commits directly on 832d5bb48, the head of feature/microvm/dev since #4775 merged, which microsoft/nvx#948 pins. It was rebased there from b92daa350: git range-diff shows the five patches unchanged, and #4775 changes none of their files.

Why

The microVM memory layout requires that loader writes and DMA ranges lie wholly inside a RAM range and cross neither the MMIO aperture nor a reserved boot structure. The Linux direct loader meets this for its own writes, but each microVM virtio device received the whole guest memory, so a descriptor failed only when its buffer left guest memory, as in the aperture, and then only for the part that did. A guest could point a buffer at the interrupt-status page at 0x30000, which the devices themselves update, or at the ISA hole. The guest can write those pages directly, so this did not weaken isolation, but the devices enforced less than the layout states.

Guest memory views (2607a7d55)

GuestMemory::restrict returns a view of a memory that admits only the accesses that lie wholly inside a set of page-aligned ranges. The view shares the regions, and so the mappings, of the memory that it restricts, so an admitted access takes the same path as before, zero-copy page locks included. GuestMemoryInner::region, which every access takes before it reaches a mapping or a fallback, refuses any other access with an out-of-range error before touching memory and calls the view's refusal hook. Reads, writes, fills, compare-exchanges, probes, page locks, IOVA lookups, and subranges all pass through it. full_mapping and sharing return None for a view. GuestMemory::check_admitted checks a range as an access would, without touching memory. Memory that is not a view pays one more None check per access.

An emulated IOMMU-style view, which translates each access through the fallback methods, would have given up the mapping and with it the block path's zero-copy locks; the bitmap feature is OpenHCL's and would have added a bitmap check to every OpenVMM access.

Whole-descriptor refusal in the queue (35af55902)

A device reads and writes a buffer in pieces, so through a view it could still touch the part of a buffer that lies inside RAM before reaching the part outside. The queue now checks each descriptor's whole buffer, and each indirect table, with check_admitted before handing the chain to the device, and refuses the chain with the new QueueError::BufferNotAdmitted. As with any malformed chain, the queue does not consume it and retires its fetch side. The rings already lie in subranges that the queue creates when it starts, which a view refuses as a whole. Memory that is not a view admits every range, so nothing changes for other devices.

MMIO transport restriction and reporting (b7bec57cd)

VirtioMmioDevice::restrict_dma hands the device a view that admits only the given ranges, through each queue's QueueResources; the transport keeps the unrestricted memory for the shared interrupt-status word.

A refusal marks the device as needing reset, as virtio 1.2 section 2.1.2 provides: the view's hook latches the refusal and wakes the transport, which sets DEVICE_NEEDS_RESET and, if the driver has set DRIVER_OK, sends a configuration-change notification, once until the next reset. A refused ring fails the queue's start and so DRIVER_OK; the transport resets the device as for any failed start but leaves DEVICE_NEEDS_RESET set, without a notification. A device reset clears the mark and any latched refusal, as does a restore, which also checks the saved rings against the view.

An error completion was the alternative, but no single form serves every device: the refused buffer may be the one that would carry the error, such as a virtio-blk status byte or a virtio-fs reply header, and virtio-net and virtio-console have none.

The e820 RAM ranges (09a68a784, bd2b27a2e)

loader::linux::microvm::mptable_ram_ranges returns the RAM entries of the map that load_x86_mptable builds, from the same build_mptable_e820, so the two cannot diverge. VirtioMmioSlots computes them for the first microVM device from the memory layout and the loader's reservations (the status page, plus the ISA hole that the loader adds) and restricts every device to them. On a snapshot restore the layout is the restored one, so the ranges include the RAM that a restore activates beyond the captured base, which the guest adds and onlines. The Linux direct-boot page of the Guide describes the restriction.

Tests

  • guestmem: test_restrict (admitted accesses, also across abutting ranges; refused accesses through every path, without writing even their admitted part; the hook; locks, subranges, and checks; no whole mapping or sharing) and test_restrict_rejects_invalid_restrictions.
  • virtio (tests::dma), with memory restricted to the RAM of a 4-GiB microVM:
    • queue_refuses_buffers_outside_ram: buffers in, or crossing into or out of, the status page, the ISA hole, and the aperture are refused once and leave the queue retired; buffers right beside them are handed over.
    • queue_refuses_indirect_tables_outside_ram: a table in the ISA hole, and a table in RAM whose buffer is the interrupt-status word.
    • queue_refuses_rings_outside_ram: a descriptor table, available ring, or used ring in or crossing into those ranges fails the queue's start.
    • mmio_device_refuses_buffers_outside_ram_and_needs_reset: through the MMIO transport with the shared-status word in the status page, the device writes none of a refused writable buffer, the transport sets DEVICE_NEEDS_RESET and publishes a configuration change, which is the only change to the status page, and a reset clears both.
    • mmio_device_writes_buffers_inside_ram and mmio_device_refuses_rings_outside_ram_and_needs_reset (DRIVER_OK fails, only DEVICE_NEEDS_RESET remains, no notification).
  • loader: mptable_ram_ranges_exclude_status_page_isa_hole_and_mmio_gap, for 8 GiB and 256 MiB.

Validation

  • On this branch, on Windows: cargo clippy --all-targets -- -D warnings and cargo doc --no-deps for guestmem (with and without bitmap), virtio, loader, and openvmm_core; cargo nextest run --profile agent for guestmem (7, and 9 with bitmap), virtio (147), loader (55), openvmm_core (118), and virtio_blk, virtio_net, virtiofs, and virtio_console (262 passed, 1 skipped); and cargo xtask fmt (clean). The second commit's tree also passes virtio's clippy and tests on its own.
  • After the rebase onto 832d5bb48: cargo xtask fmt, cargo clippy --all-targets -- -D warnings for guestmem, virtio, loader, virtiofs, and openvmm_core, and cargo nextest run --profile agent for guestmem, virtio, loader, and virtiofs (322) and openvmm_core (118), all clean.
  • NVX's microVM scenarios, which boot guests that use every microVM virtio device: with the same patches on b92daa350 (the previous head, d647d4de4), openvmm: refuse virtio device buffers outside the e820 RAM ranges nvx#974's CI passed every microVM suite on Linux/KVM, Linux/MSHV, and Windows/WHP, as well as OpenVMM's unit tests and VMM tests on those platforms. The NVX pull request will pin this head once openvmm: let a platform restore attach an aggregate to a dormant filesystem slot nvx#948 moves NVX's pin to 832d5bb48.
  • Not run locally: the VMM tests, which do not use the microVM profile.

Notes

  • Guests that keep their buffers in RAM see no change. A refusal leaves the refused queue stalled, as any malformed chain does; a driver recovers by resetting the device.
  • Only the microVM profile restricts device DMA. Other devices and transports keep the whole guest memory.

@github-actions github-actions Bot added Guide unsafe Related to unsafe code labels Oct 28, 2026
@github-actions

github-actions Bot commented Oct 28, 2026

Copy link
Copy Markdown

⚠️ Unsafe Code Detected

This PR modifies files containing unsafe Rust code. Extra scrutiny is required during review.

For more on why we check whole files, instead of just diffs, check out the Rustonomicon

@ppenna Pedro Henrique Penna (ppenna) mentioned this pull request Oct 28, 2026
Merged
@ppenna
guestmem: add views that admit only a set of ranges
2607a7d 
A device reaches guest memory through the GuestMemory that its transport
gives it, and nothing narrows what that GuestMemory reaches. A microVM
virtio device gets the whole guest memory, so it reads or writes any
range that the guest names in a descriptor as long as memory backs it,
including the ranges that the e820 map reserves inside RAM, such as the
shared virtio interrupt-status page at 0x30000 and the ISA hole
(microsoft/nvx#554). An emulated IOMMU narrows a device's view by
translating each access through the fallback methods, but that gives up
the mapping, and with it the zero-copy page locks on which the block
path relies.

Add GuestMemory::restrict, which returns a view of a memory that admits
only the accesses that lie wholly inside a set of page-aligned ranges.
The view shares the regions, and so the mappings, of the memory that it
restricts, and keeps that memory alive, so an admitted access takes the
same path as through the memory itself. GuestMemoryInner::region, which
every access takes before it reaches a mapping or a fallback, refuses
any other access with an out-of-range error before touching memory, and
calls the view's refusal hook, if any, with the refused address and
length. Reads, writes, fills, compare-exchanges, probes, page locks,
IOVA lookups, and subranges all pass through it, and a subrange of a
view stays inside a range that the view admitted when it was created.
full_mapping and sharing return None for a view, since either would
expose all of the memory.

A view admits the union of its ranges, merging overlapping and abutting
ones. The ranges must be page-aligned, because a page lock admits a page
by its first byte. An empty access touches nothing and is admitted
anywhere. A view of a view is rejected rather than intersected, as
nothing needs one.

Add GuestMemory::check_admitted, which checks a range as an access
would, refusal hook included, without touching memory, so that a caller
can refuse a request as a whole before acting on any part of it. Memory
that is not a view admits every range.

Memory that is not a view pays one more None check per access. An
access through a view also searches its ranges, of which a microVM has
four.

The tests check that a view admits accesses inside its ranges, even
across abutting ones; that it refuses, through every access path and
without writing even the part inside its ranges, an access that lies in
or crosses into a page outside them; that its hook sees each refusal;
that the memory it restricts still admits everything; and that
unaligned ranges and a view of a view are rejected.

Part of microsoft/nvx#554.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
virtio: refuse a descriptor whose buffer the device memory does not a…
35af559 
…dmit

A virtio queue hands a device each descriptor's address and length
without checking them, and the device reads and writes the buffer later,
often in pieces: a header before the data, the data page by page, or
just a prefix when the buffer is longer than what the device needs.
Through a restricted GuestMemory each of those accesses fails on its
own, so the device could still read or write the part of a buffer that
lies inside the ranges that the memory admits before reaching the part
outside them.

Check each descriptor's whole buffer, and each indirect table, with
GuestMemory::check_admitted before handing the chain to the device, and
refuse the chain with the new QueueError::BufferNotAdmitted when the
memory does not admit it. As with any other malformed chain, the queue
does not consume it and retires its fetch side, so the device reads and
writes none of the chain and sees the error once. The queue already
places each ring in a subrange that it creates when it starts, which a
restricted memory refuses as a whole. Memory that is not restricted
admits every range, so nothing changes for a device that gets one.

The tests restrict a queue's memory to the RAM ranges of the e820 map of
a microVM with 4 GiB of RAM. They check that the queue refuses buffers
in the shared interrupt-status page, in the ISA hole, and in the MMIO
aperture, and buffers that cross into or out of them, reporting each
refusal once and staying retired afterward; that it hands over buffers
that end or begin right beside them; that it refuses an indirect table
in the ISA hole and a table in RAM whose buffer is the interrupt-status
word; and that it fails to start when a ring lies in or crosses into one
of them.

Part of microsoft/nvx#554.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
virtio: let an MMIO transport restrict device DMA and report refusals
b7bec57 
With guest memory views and the queue's admission check, a device can
be kept to a set of ranges of guest memory, but no transport gives a
device such a view, and nothing tells the guest when the device refuses
a ring or a buffer.

Add VirtioMmioDevice::restrict_dma, which a VMM calls before the guest
starts the device. It replaces the memory that the transport hands the
device in each queue's QueueResources with a view that admits only the
given ranges. The transport keeps the unrestricted memory for itself,
so a shared interrupt-status word outside the ranges still works.

Report a refusal by marking the device as needing reset, as virtio 1.2
section 2.1.2 provides for a device that enters an error state from
which only a reset recovers it. The view's hook latches the refusal and
wakes the transport, which sets DEVICE_NEEDS_RESET in the device status
and, if the driver has set DRIVER_OK, sends a configuration-change
notification, once until the next reset. A refused ring fails the
queue's start and so DRIVER_OK; the transport resets the device as for
any failed start, but leaves DEVICE_NEEDS_RESET set, without a
notification, since DRIVER_OK is then clear. A device reset clears the
mark and any refusal latched before it, as does a restore, which also
checks the saved rings against the view.

An error completion was the alternative, but no single form serves
every device: the refused buffer may be the one that would carry the
error, such as a virtio-blk status byte or a virtio-fs reply header,
and virtio-net and virtio-console have no error completion at all.
DEVICE_NEEDS_RESET reports a refusal the same way for every device, and
the queue has already stopped serving the refused chain.

The tests restrict an MMIO device's DMA to the RAM ranges of the e820
map of a microVM with 4 GiB of RAM, with its shared interrupt-status
word in the status page, as the microVM places it. They check that the
device refuses a writable buffer in, or crossing into, the
interrupt-status page, the ISA hole, or the aperture: it writes none of
the buffer, the transport marks it as needing reset and notifies the
driver through the interrupt-status word, which the device leaves
otherwise untouched, and a reset clears both. They check that the
device writes buffers right beside those ranges without being marked;
and that a ring in or crossing into them fails DRIVER_OK, leaving only
DEVICE_NEEDS_RESET set and no notification, until a reset.

Part of microsoft/nvx#554.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
loader: return the RAM ranges of the microVM e820 map
09a68a7 
The microVM MP-table loader builds the e820 map from the memory layout:
it reserves the ISA hole and the ranges that the VMM passes, such as the
shared virtio interrupt-status page, and leaves the MMIO aperture out of
RAM. Only the loader knows the resulting RAM ranges, so a VMM that must
keep device DMA inside the RAM that the guest sees would have to repeat
the loader's reservations.

Add mptable_ram_ranges, which returns the RAM entries of the map that
load_x86_mptable builds for a layout and its reserved ranges. It derives
them from the same build_mptable_e820, so the two cannot diverge, and
validates the layout and reservations as loading does.

The test checks the ranges for 8 GiB of RAM, which the aperture splits,
and for 256 MiB, which ends below it: both leave out the status page and
the ISA hole, and neither reaches into the aperture.

Part of microsoft/nvx#554.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
microVM: restrict virtio device DMA to the RAM ranges of the e820 map
bd2b27a 
The microVM layout requires that DMA ranges lie wholly inside a RAM
range of the e820 map and cross neither the MMIO aperture nor a reserved
boot structure. Each microVM virtio device received the whole guest
memory, though, so a descriptor failed only when its buffer left guest
memory, as in the aperture, and then only for the part that did. A guest
could point a buffer at a range that the e820 map reserves inside RAM,
such as the shared virtio interrupt-status page at 0x30000, which the
devices themselves update, or the ISA hole, and the device would read
or write it (microsoft/nvx#554). The guest can write those pages
directly, so this did not weaken isolation, but the devices enforced
less than the layout states.

Restrict the DMA of every microVM virtio device to the RAM ranges of
the e820 map, which mptable_ram_ranges derives from the memory layout
and the same reservations that the loader uses: the status page, and
the ISA hole, which the loader adds. The devices thus refuse a queue
whose rings, or a descriptor whose buffer, lies even partly outside
RAM: they read and write none of it, and each refusal marks the device
as needing reset. The transports keep the whole guest memory for the
shared interrupt-status words that they update.

The ranges come from the layout at device construction, which on a
snapshot restore is the restored layout, so they include the RAM that a
restore activates beyond the captured base, which the guest adds and
onlines. They are computed for the first microVM device, as the e820
map applies only to the microVM profile.

The Linux direct-boot page of the Guide describes the restriction.

Part of microsoft/nvx#554.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@ppenna
Pedro Henrique Penna (ppenna) force-pushed the copilot/microvm-virtio-ram-only-dma branch from d647d4d to bd2b27a Compare October 29, 2026 00:09
Pedro Henrique Penna (ppenna) added a commit to microsoft/nvx that referenced this pull request Oct 29, 2026
@ppenna
openvmm: refuse virtio device buffers outside the e820 RAM ranges
677fe92 
Pin microsoft/openvmm#4794 at bd2b27a2ef9339bd16692c93facb8d84952dc9f8,
five commits on 832d5bb48, the previous pin, which is the head of
feature/microvm/dev.

The microVM memory layout requires that DMA ranges lie wholly inside a
RAM range of the e820 map and cross neither the MMIO aperture nor a
reserved boot structure, but each microVM virtio device received the
whole guest memory. A descriptor failed only when its buffer left guest
memory, as in the aperture, and then only for the part that did, so a
guest could point a device buffer at the shared virtio interrupt-status
page at 0x30000, which the devices themselves update, or at the ISA
hole. The guest can write those pages directly, so this did not weaken
isolation, but the devices enforced less than the layout states.

The first commit adds guest memory views, which admit only the accesses
that lie wholly inside a set of page-aligned ranges and refuse any
other before touching memory, on every access path, page locks
included, while sharing the mappings of the memory that they restrict,
so that admitted accesses keep their zero-copy paths.

The second makes a virtio queue check each descriptor's whole buffer,
and each indirect table, against the device's memory before handing the
chain to the device, and refuse the chain as malformed when the memory
does not admit it, so that a device reads and writes none of a buffer
that lies even partly outside the admitted ranges. Rings already lie in
subranges that the memory refuses as a whole when a queue starts.

The third lets an MMIO transport restrict its device to a set of ranges
and report each refusal by setting DEVICE_NEEDS_RESET, with a
configuration-change notification once the driver has set DRIVER_OK,
as virtio 1.2 provides for a device in an error state. A refused ring
fails DRIVER_OK and leaves DEVICE_NEEDS_RESET alone set. A reset clears
the mark. The transport keeps the whole memory for the shared
interrupt-status word.

The fourth returns the RAM ranges of the microVM e820 map from the
loader that builds it, and the fifth restricts every microVM virtio
device to them: the layout's RAM, which after a restore includes the
RAM that the restore activates, less the interrupt-status page and the
ISA hole.

The protocol vectors and built-in CPU profiles that nvx-rs copies are
unchanged.

Refs #554

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pedro Henrique Penna (ppenna) added a commit to microsoft/nvx that referenced this pull request Oct 29, 2026
@ppenna
doc: map the device DMA restriction in the OpenVMM roadmap
5d91773 
Record the five commits that the previous commit pins as positions
355-359 of the microVM chain, and move the NVX pin to bd2b27a2e.

Positions 355-359 restrict the DMA of the microVM virtio devices to the
RAM ranges of the e820 map: guest memory views, the virtio queue's
whole-buffer check, the MMIO transport's restriction and needs-reset
report, the loader's e820 RAM ranges, and the microVM worker that
applies them. Position 355 changes only guestmem, which no other commit
of the chain changes, so it can go upstream on its own, as positions
350-352 can, and position 356 adds only the queue's check on top of it.
Positions 357-359 build on the fork's shared interrupt-status transport,
MP-table loader, and microVM worker; say so in the table's remarks. The
pin is now 202 commits past the mapped tip.

Positions 353 and 354 are no longer only a proposal:
microsoft/openvmm#4775 merged them, so position 354, 832d5bb48, is the
head of feature/microvm/dev. Say so in the scope, and name
microsoft/openvmm#4794 as the proposal of positions 355-359.

Refs #554

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Oct 29, 2026

Copy link
Copy Markdown

4 Petri tests failed (3 unstable)

@ppenna
Pedro Henrique Penna (ppenna) merged commit bd2b27a into feature/microvm/dev Oct 29, 2026
92 of 108 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Guide unsafe Related to unsafe code

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ppenna