LibVMI on VirtualBox: default access method, and revived libvmi libmicrovmi branch #312

Problem Statement

LibVMI users can't introspect VirtualBox guests. The Wenzel/libvmi libmicrovmi branch, which routes LibVMI's hypervisor access through libmicrovmi, dates from 2021: it targets the microvmi v0.3.1 C API, is 113 commits behind upstream LibVMI, and only maps the Xen and KVM Drivers to a LibVMI mode. Now that libmicrovmi's VirtualBox Driver works again (dev, with the VBoxVMI method and the MachineDebugger method), LibVMI cannot reach it, and nobody has checked that LibVMI's memory exploration (process listing through an OS profile) works on top of either access method.

On the libmicrovmi side, a caller must always name an Access method (ADR-0003), so every LibVMI user would have to know whether the VM serves a VBoxVMI socket. There is also no way to ask libmicrovmi for a guest's vCPU count through the C API.

Solution

1. libmicrovmi (dev): when no Access method is selected, the VirtualBox Driver uses the VBoxVMI method at the VM's default socket, and falls back to the MachineDebugger method only when that socket does not exist. An explicitly selected method never falls back. The VBoxVMI socket path becomes optional (default path when omitted). The C API gains a vCPU count call. ADR-0004 records the reversal of ADR-0003's "no default access method" rule.
2. LibVMI fork: the libmicrovmi branch is rewritten as a fresh commit series on upstream libvmi/libvmi master. libmicrovmi replaces LibVMI's whole hypervisor access layer. LibVMI gains a VirtualBox LibVMI mode and init data to select the access method and socket path explicitly.
3. An end-to-end script runs a set of LibVMI examples against an Ubuntu VirtualBox guest under the default, explicit VBoxVMI and explicit MachineDebugger selections, and checks their output against the guest.

User Stories

1. As a LibVMI user, I want to vmi_init a VirtualBox VM by name, so that I can introspect VirtualBox guests with the LibVMI tools I already use.
2. As a LibVMI user, I want LibVMI to use the fast VBoxVMI method automatically when my VM serves a VBoxVMI socket, so that I get the best performance without configuration.
3. As a LibVMI user on a VirtualBox without VBoxVMI enabled, I want LibVMI to fall back to the MachineDebugger method, so that introspection still works on that VM.
4. As a LibVMI user, I want to select the MachineDebugger method explicitly, so that I can force the public-API path even when a VBoxVMI socket exists.
5. As a LibVMI user, I want to select the VBoxVMI method explicitly, so that I get an error instead of a silent slow fallback when VBoxVMI is not available.
6. As a LibVMI user, I want to pass an explicit VBoxVMI socket path, so that I can reach a VM whose socket lives somewhere other than the default.
7. As a LibVMI user, I want an explicit VBoxVMI selection without a socket path to use the VM's default socket, so that I don't have to compute the VM UUID path myself.
8. As a LibVMI user, I want vmi_get_access_mode to report a VirtualBox LibVMI mode, so that my tool can tell it is talking to VirtualBox.
9. As a LibVMI user, I want vmi_get_num_vcpus to return the guest's real vCPU count, so that per-vCPU loops cover every vCPU.
10. As a LibVMI user, I want to read vCPU registers (CR3, RIP, IDTR/GDTR, ...) of each vCPU, so that I can translate addresses and inspect execution state.
11. As a LibVMI user, I want vmi_get_max_physical_address to return the guest's max physical address (top of RAM ranges), so that memory dumps cover the MMIO hole correctly.
12. As a LibVMI user, I want to read guest physical memory, so that address translation and OS-level introspection work.
13. As a LibVMI user, I want to write guest physical memory, so that LibVMI's write API keeps working after the native drivers are gone.
14. As a LibVMI user, I want to pause and resume the VM, so that I can take consistent snapshots of guest state.
15. As a LibVMI user, I want vmi-process-list to list the Ubuntu guest's processes through an ISF profile, so that I know memory exploration is consistent with what the guest sees.
16. As a LibVMI user, I want identical vmi-process-list output over VBoxVMI and MachineDebugger, so that the choice of access method doesn't change results.
17. As a LibVMI user, I want the event API to fail cleanly with VMI_FAILURE, so that tools relying on VMI events fail loudly instead of hanging.
18. As a LibVMI user of the Xen and KVM LibVMI modes, I want those modes to keep their existing mapping through libmicrovmi (including the KVMI socket init data), so that the branch doesn't regress them.
19. As a LibVMI user selecting the file or bareflank LibVMI mode, I want vmi_init to fail as unsupported, so that I know libmicrovmi has no Driver for them.
20. As a libmicrovmi Rust caller, I want omitting the VirtualBox init params to select the default access method, so that I don't need to know whether VBoxVMI is enabled on the VM.
21. As a libmicrovmi C caller, I want access_method = 0 to select the default access method, so that a zero-initialised params struct does the right thing.
22. As a libmicrovmi C caller, I want a vCPU count call, so that I can size per-vCPU work without guessing.
23. As a libmicrovmi Python caller, I want omitting the VirtualBox params to select the default access method, so that the bindings behave like the Rust API.
24. As a libmicrovmi caller, I want the default to fall back only when the VBoxVMI socket file is missing, so that a busy socket (another client attached), a permission rejection, a protocol mismatch or a handshake timeout is reported instead of hidden behind a slow fallback.
25. As a libmicrovmi caller, I want an info log line naming the access method the default picked (and why it fell back), so that I can tell which path a run took.
26. As a libmicrovmi caller with only one VirtualBox access method compiled in, I want selection to behave predictably: VBoxVMI-only requires a socket path; MachineDebugger-only uses MachineDebugger; the default with fallback exists only when both are compiled in.
27. As a libmicrovmi maintainer, I want the default-socket lookup to resolve the VM name to its UUID and honour the VM's VBoxInternal/VMI/SocketPath override through the VirtualBox API, so that the default matches where the patched VirtualBox actually listens.
28. As a libmicrovmi maintainer, I want ADR-0004 to supersede ADR-0003's "no default access method" rule, so that future readers know why the default came back.
29. As the libvmi fork maintainer, I want the libmicrovmi branch rebuilt as a short fresh series on upstream master and force-pushed, so that it is reviewable and rebaseable again.
30. As the libvmi fork maintainer, I want the fork's master fast-forwarded to upstream master, so that the branch's base is visible on the fork.
31. As the libvmi fork maintainer, I want CI to build libmicrovmi dev from a pinned commit, then LibVMI and its examples, so that the branch's CI is green and catches build breaks.
32. As the libvmi fork maintainer, I want the README to say how to install libmicrovmi and select a VirtualBox access method, so that users can reproduce the setup.
33. As a developer, I want a reproducible microvmi-ubuntu test guest (Ubuntu 24.04 server installed to disk, 2 vCPUs, 2 GiB, kernel held, base snapshot at the login prompt), so that end-to-end runs are deterministic.
34. As a developer, I want an ISF profile built with dwarf2json from the guest kernel's dbgsym package, plus the matching libvmi.conf entry, stored next to the VirtualBox dev harness, so that vmi-process-list runs without manual setup.
35. As a developer, I want one end-to-end script that restores the snapshot and runs each example under each access method selection, so that one command shows whether LibVMI on VirtualBox works.

Implementation Decisions

libmicrovmi (dev, one PR, lands first)

• VirtualBox init params: "no VirtualBox params" now selects the default access method instead of being an error. The VBoxVMI variant's socket path becomes optional; omitted means the VM's default socket. There is no new Auto variant.

• C ABI keeps its shape (access_method plain integer + socket_path); only the meaning of values changes:

  access_method	socket_path	meaning
  0	any	default: VBoxVMI at default socket; fall back to MachineDebugger only if that socket file is missing
  MachineDebugger (1)	ignored	MachineDebugger, never falls back
  VBoxVMI (2)	set	VBoxVMI at that path, never falls back
  VBoxVMI (2)	null	VBoxVMI at default socket path, never falls back (previously an error)

• Default socket path: <VirtualBox user home>/vmi/<VM UUID>.sock, overridden by the VM's VBoxInternal/VMI/SocketPath extradata; name→UUID and extradata are resolved through the VirtualBox API. So the default (and VBoxVMI without a path) is only available when both access methods are compiled in (the virtualbox feature).

• Fallback trigger: only "socket file does not exist". Every other VBoxVMI failure is returned unchanged.

• The Driver logs the chosen access method at info level, including the fallback reason. No API to query the chosen method.

• New C export returning the guest vCPU count (the Rust API already has it).

• Python bindings: omitted VirtualBox params mean the default.

• ADR-0004 supersedes the "no default access method" part of ADR-0003.

• Glossary already updated (PR docs: flag LibVMI mode vs access method ambiguity #311): LibVMI mode, and the default-selection relationship.

LibVMI fork (libmicrovmi branch, rewritten)

• Base: upstream libvmi/libvmi master; fork master fast-forwarded to it; branch force-pushed, old history discarded.
• Fresh series, roughly: build system finds/links an installed libmicrovmi → remove native drivers (Xen, KVM, file, bareflank) → libmicrovmi driver (init/destroy, physical read/write, pause/resume, vCPU registers, vCPU count, max physical address, memory cache) → VirtualBox LibVMI mode and init data → examples → README.
• libmicrovmi replaces the whole hypervisor access layer. Existing wiring from the old branch stays: memory cache, IDTR/GDTR register mapping, Xen/KVM driver type → LibVMI mode mapping, KVMI socket init data → libmicrovmi KVM Unix socket param.
• vmi_mode keeps every upstream value; adds VMI_VIRTUALBOX. Driver type mapping: Xen → Xen, KVM → KVM, VirtualBox → VirtualBox, memflow → a memflow LibVMI mode. File and bareflank fail at init as unsupported by libmicrovmi.
• New init data: one selecting the VirtualBox access method (MachineDebugger or VBoxVMI), one carrying a VBoxVMI socket path. Neither given → access_method = 0 (default). These map one-to-one onto the libmicrovmi C ABI above.
• Write path goes through libmicrovmi's physical write.
• Event API returns VMI_FAILURE (VMI events are Phase 2).
• libmicrovmi is consumed as a system install of dev built with the virtualbox feature.
• CI: build libmicrovmi dev from a pinned commit, install it, build LibVMI and the examples; build-only.

Examples

• Kept from the old branch: pause example, map-paddr.
• Upstream: vmi-dump-memory, rwmem, vmi-process-list.
• New small example: prints vCPU count, max physical address, and CR3/RIP per vCPU.

Testing Decisions

• A good test drives the public API and checks externally visible results against the guest; it does not inspect driver internals.
• One seam for LibVMI: its public API, exercised through the example binaries against microvmi-ubuntu. No new unit tests inside LibVMI and no seam inside its libmicrovmi layer.
• The end-to-end script lives with the VirtualBox dev harness (next to vm.sh), not in the libvmi fork. Like libmicrovmi's VirtualBox integration harness, it is configured from the environment (VM, snapshot) and restores the base snapshot before every run.
• Each example runs three ways: default selection, explicit VBoxVMI, explicit MachineDebugger. Default selection runs once with VBoxVMI enabled on the VM (log must say VBoxVMI) and once with it disabled (log must say MachineDebugger). One run with an explicit socket path that does not exist must fail.
• Pass criteria:
  • vCPU info: count is 2; CR3/RIP non-zero and differ across vCPUs; max physical address reported.
  • pause example: guest pauses and resumes.
  • map-paddr: a physical page reads.
  • vmi-dump-memory: dump size matches the max physical address.
  • rwmem: a value written to a free physical page reads back (snapshot restored afterwards).
  • vmi-process-list: PID/name list matches ps -e in the guest, and is identical over both access methods.
• libmicrovmi: extend its existing VirtualBox integration harness (VBOX_TEST_METHOD-driven, snapshot restore per test) with default-selection cases: VBoxVMI disabled → MachineDebugger; enabled → VBoxVMI; busy socket → error, not fallback; explicit missing socket → error. Prior art: the existing VirtualBox integration and error-path tests.
• Xen/KVM paths: not tested here; existing wiring kept as is.

Out of Scope

• VMI events through LibVMI (Phase 2).
• Testing Xen or KVM through the branch.
• Memflow connector arguments from LibVMI.
• OS-aware examples other than vmi-process-list (module list, dmesg, Windows guests).
• Upstreaming the branch to libvmi/libvmi.
• Windows hosts.
• A Debian package of libmicrovmi dev.

Further Notes

• Ordering: libmicrovmi PR to dev first; the libvmi branch pins the resulting dev commit (locally and in CI).
• Risk: the Ubuntu 24.04 kernel dbgsym package must be available on ddebs and dwarf2json must handle it; the guest kernel is held so the profile stays valid.
• Risk: the old branch is discarded with no backup tag.
• The microvmi-alpine guest remains for libmicrovmi's own tests.