Replace FlatBuffers with serde/postcard for host-guest messages

.github/copilot-instructions.md

@@ -119,7 +119,6 @@ When reviewing a diff (your own or a PR), weight these for Hyperlight:
 * `src/hyperlight_guest_capi/`: guest C library.
 * `src/hyperlight_host/`: hyperlight-host library.
 * `src/hyperlight_testing/`: shared test code.
-* `schema/`: flatbuffer schemas.
 * `tests/`: test guest code in C and Rust.
 * `dev/`: development scripts and tools.
 * `fuzz/`: fuzzing tests.

.github/workflows/DailyArm64.yml

@@ -86,7 +86,7 @@ jobs:
 
   # Fuzz on aarch64. This coverage was removed from PRs in #1594 to conserve the
   # limited arm64 runners, and runs here daily instead. The tracing
-  # fuzzers (fuzz_guest_trace, fuzz_guest_estimate_trace_event) are x86_64-only
+  # fuzzers (fuzz_guest_trace, fuzz_guest_trace_decode) are x86_64-only
   fuzzing:
     needs: build-guests
     strategy:

.github/workflows/Fuzzing.yml

@@ -22,7 +22,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        target: ['fuzz_host_print', 'fuzz_guest_call', 'fuzz_host_call', 'fuzz_guest_estimate_trace_event', 'fuzz_guest_trace', 'fuzz_virtq_malformed', 'fuzz_virtq_roundtrip']
+        target: ['fuzz_host_print', 'fuzz_guest_call', 'fuzz_host_call', 'fuzz_guest_trace_decode', 'fuzz_guest_trace', 'fuzz_virtq_malformed', 'fuzz_virtq_roundtrip', 'fuzz_control_decode']
     uses: ./.github/workflows/dep_fuzzing.yml
     with:
       target: ${{ matrix.target }}

.github/workflows/ValidatePullRequest.yml

@@ -259,7 +259,7 @@ jobs:
     if: ${{ !cancelled() && !failure() }}
     strategy:
       matrix:
-        target: ['fuzz_host_print', 'fuzz_guest_call', 'fuzz_host_call', 'fuzz_guest_estimate_trace_event', 'fuzz_guest_trace', 'fuzz_virtq_malformed', 'fuzz_virtq_roundtrip']
+        target: ['fuzz_host_print', 'fuzz_guest_call', 'fuzz_host_call', 'fuzz_guest_trace_decode', 'fuzz_guest_trace', 'fuzz_virtq_malformed', 'fuzz_virtq_roundtrip', 'fuzz_control_decode']
         arch:
           - X64
         # arm64 fuzzing runs on the daily schedule (DailyArm64.yml) instead of on

CHANGELOG.md

@@ -7,6 +7,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 ### Added
 * Add `Sandbox::recreate()`, which rebuilds a sandbox's VM and memory from a snapshot. It recovers unrecoverable sandboxes and keeps configuration, host functions, and interrupt handles by @ludfjig in https://github.com/hyperlight-dev/hyperlight/pull/1751
 
+### Changed
+* **Breaking:** Encode function calls with serde/postcard instead of FlatBuffers.
+
 ### Fixed
 * Fix restore on WHP not rewinding the guest TSC on hosts with `TSC_ADJUST` support.
 

Justfile

@@ -235,10 +235,11 @@ like-ci config=default-target hypervisor="kvm":
     just fuzz-like-ci fuzz_host_print {{config}} {{hypervisor}}
     just fuzz-like-ci fuzz_guest_call {{config}} {{hypervisor}}
     just fuzz-like-ci fuzz_host_call {{config}} {{hypervisor}}
-    just fuzz-like-ci fuzz_guest_estimate_trace_event {{config}} {{hypervisor}}
+    just fuzz-like-ci fuzz_guest_trace_decode {{config}} {{hypervisor}}
     just fuzz-like-ci fuzz_guest_trace {{config}} {{hypervisor}}
     just fuzz-like-ci fuzz_virtq_malformed {{config}} {{hypervisor}}
     just fuzz-like-ci fuzz_virtq_roundtrip {{config}} {{hypervisor}}
+    just fuzz-like-ci fuzz_control_decode {{config}} {{hypervisor}}
 
     @# spelling
     typos
@@ -666,21 +667,6 @@ coverage-ci hypervisor="kvm": (coverage-run hypervisor)
     cargo +{{nightly-toolchain}} llvm-cov report --lcov --output-path target/coverage/lcov.info
     cargo +{{nightly-toolchain}} llvm-cov report | tee target/coverage/summary.txt
 
-###################
-### FLATBUFFERS ###
-###################
-
-gen-all-fbs-rust-code:
-    flatc --rust --rust-module-root-file --gen-all -o ./src/hyperlight_common/src/flatbuffers/ ./src/schema/all.fbs
-    just fmt-apply
-
-install-vcpkg:
-    cd .. && git clone https://github.com/Microsoft/vcpkg.git || cd -
-    cd ../vcpkg && ./bootstrap-vcpkg{{ bin-suffix }} && ./vcpkg integrate install || cd -
-
-install-flatbuffers-with-vcpkg: install-vcpkg
-    cd ../vcpkg && ./vcpkg install flatbuffers || cd -
-
 ###################################
 ### SNAPSHOT GOLDEN HELPERS     ###
 ###################################

README.md

@@ -119,7 +119,6 @@ Or skip setup entirely with a codespace:
 | [src/hyperlight_component_macro](./src/hyperlight_component_macro) | Proc macros for WIT-based host/guest bindings                                        |
 | [src/hyperlight_component_util](./src/hyperlight_component_util)   | Shared implementation for WIT binding generation                                     |
 | [src/hyperlight_testing](./src/hyperlight_testing)                 | Shared test utilities                                                                |
-| [src/schema](./src/schema)                                         | FlatBuffer schema definitions                                                        |
 | [src/trace_dump](./src/trace_dump)                                 | Tool for dumping and visualizing trace data                                          |
 | [src/tests](./src/tests)                                           | Test guest programs (Rust and C)                                                     |
 

docs/README.md

@@ -38,7 +38,6 @@ This project is composed internally of several components, depicted in the below
 * [Virtqueue host and guest communication](./virtio-host-guest-communication.md)
 * [Component wire format](./component-wire-format.md)
 * [How to debug a Hyperlight guest](./how-to-debug-a-hyperlight-guest.md)
-* [How to use Flatbuffers in Hyperlight](./how-to-use-flatbuffers.md)
 * [How to make a Hyperlight release](./how-to-make-releases.md)
 * [Getting Hyperlight Metrics, Logs, and Traces](./hyperlight-metrics-logs-and-traces.md)
 * [Benchmarking Hyperlight](./benchmarking-hyperlight.md)

docs/security-guidance-for-developers.md

@@ -9,7 +9,7 @@ This document discusses the security requirements and best practices for service
 ## Brief Checklist
 * All host functions that receive parameters from a guest, or operate indirectly on guest data _MUST_ be continuously fuzzed
 * Host functions _MUST NOT_ call APIs or be used to expose functionality deemed risky in a multi-tenant context
-* Guests and host processes _MUST_ use the same version of a FlatBuffer definition
+* Guests and host processes _MUST_ use compatible Hyperlight versions
 
 More detailed guidance on the requirements and best practices is detailed below.
 
@@ -43,24 +43,13 @@ In a multi-tenant context, the following operations are considered security sens
 
 If any of these operations is performed in a host process, a security audit _MUST_ occur.
 
-## Flatbuffers - Guests and host processes _MUST_ use compatible versions of a FlatBuffer definitions
+## Guests and host processes _MUST_ use compatible Hyperlight versions
 
-The guests and host processes _MUST_ use the exact same versions of a FlatBuffer definition. I.e., the same .fbs file _MUST_ be used for generating the encoders and decoders.
+Host and guest exchange [postcard](https://docs.rs/postcard) messages built from Rust types in `hyperlight-common` with [serde](https://serde.rs). The format has no schema evolution. Guests _MUST_ be built against a `hyperlight-guest` version that matches the host.
 
-## Flatbuffers - If using the same language for development, the guests and host processes _SHOULD_ use the same version of flatc compilers.
+## Messages from the guest _MUST_ be copied and validated before use
 
-This can be seen in the header files containing FLATBUFFERS_COMMON_READER_H. For instance:  `/* Generated by flatcc 0.6.2 FlatBuffers schema compiler`.
-We emit this recommendation because there is a history of compiler bugs, which may affect certain behaviors (encoding, decoding). We emit this recommendation because there is a history of compiler bugs, which may adversely affect certain behaviors (encoding, decoding).
-
-## Flatbuffers – a verifier should always be called before any decoder. In the case of failed verification, the input _MUST NOT_ be processed.
-
-For Rust code, if the return code is InvalidFlatBuffer, the input _MUST_ be rejected.
-
-## Flatbuffers – the host process _MUST NOT_ operate on Flatbuffers from several threads.
-
-Because of the zero-copy approach that FlatBuffers is using, there is a risk of memory safety issues. Flatbuffers are unsafe to be used in a multithreaded environment. This is explicitly indicated in several parts of the Flatbuffer documentation.
-
-Additionally, because Flatbuffers tainted data coming from the guests, this is even more critical in a multi-tenant scenario.
+The host copies guest data out of shared memory before decoding it. Decoding _MUST_ reject malformed input, trailing bytes, and lengths beyond the received data. Rejected input _MUST NOT_ be processed.
 
 ## A rust host process _SHOULD_ handle [panics](https://doc.rust-lang.org/book/ch09-03-to-panic-or-not-to-panic.html) or the service _SHOULD_ restart automatically
 

docs/security.md

@@ -12,7 +12,7 @@ Hyperlight runs all guest code inside a Virtual Machine, Each VM only has access
 
 ### Host-Guest Communication (Serialization and Deserialization)
 
-All communication between the host and the guest is done through a shared memory buffer. Messages are serialized and deserialized using [FlatBuffers](https://flatbuffers.dev/). To minimize attack surface area, we rely on FlatBuffers to formally specify the data structures passed to/from the host and guest, and to generate serialization/deserialization code. Of course, a compromised guest can write arbitrary data to the shared memory buffer, but the host will not accept anything that does not match our strongly typed FlatBuffer [schemas](../src/schema).
+All communication between the host and the guest is done through a shared memory buffer. Messages are serialized with [serde](https://serde.rs) into the [postcard](https://docs.rs/postcard) format, using strongly typed Rust definitions in `hyperlight-common`. The host copies guest data out of shared memory before decoding it. Decoding rejects malformed input and trailing bytes. Every length is checked against the received message, so allocations stay proportional to its size. A compromised guest can write arbitrary data to the shared memory buffer, but the host accepts only messages that decode into these types.
 
 ### Accessing host functionality from the guest
 

docs/snapshot-versioning.md

@@ -52,6 +52,9 @@ regenerated.
 ABI 9 removes the heap region from snapshot memory. Earlier snapshots must
 be regenerated.
 
+ABI 10 encodes control data with postcard and carries its length in a 16-byte
+message header. Earlier snapshots must be regenerated.
+
 The `OCI_LAYOUT_VERSION` constant is pinned by the OCI image-layout
 spec at `1.0.0`.
 
@@ -74,7 +77,7 @@ them.
 
 ## Enforcement
 
-The format is large and easy to change by accident. Two mechanisms
+The format is large and easy to change by accident. Three mechanisms
 catch a change to it so reviewers do not have to spot every break by
 eye, and so a developer who breaks the format unintentionally finds
 out at build time rather than in production.
@@ -87,6 +90,12 @@ OCI layout version, the `HyperlightPEB` size, every `OutBAction` and
 `VmAction` discriminant, and `BASE_ADDRESS`. If the source value
 drifts from the copy in `tripwires.rs`, the crate fails to compile.
 
+Wire format tests in
+[src/hyperlight_common/src/flatbuffer_wrappers/abi_tests.rs](../src/hyperlight_common/src/flatbuffer_wrappers/abi_tests.rs)
+pin the encoded bytes of every host-guest message: the message header,
+function calls and results, guest logs and trace events. Snapshots
+contain guest code built against these encodings.
+
 The snapshot golden verify test
 (`cargo test -p hyperlight-host --test snapshot_goldens`) loads
 snapshots from a local directory (populated by `just snapshot-goldens-pull`,