Repository navigation
Conversation
This comment has been minimized.
This comment has been minimized.
c27b99f to
c056ede
Compare
This comment has been minimized.
This comment has been minimized.
c056ede to
e474c07
Compare
This comment has been minimized.
This comment has been minimized.
|
Nice! I believe all the: |
Thanks nice catch. They should go, but i'd prefer do it in a follow up pr, it brings a lot of unrelated code changes if you don't mind and would be a breaking change (they're in public api |
There was a problem hiding this comment.
🟡 Changes recommended
The security documentation overstates allocation guarantees because decoded parameters and trace fields lack per-collection count limits despite transport-size bounds.
1 open issue
What changed in this PR
This PR implements HIP 0008 by using serde/postcard for Hyperlight’s host-guest messages while keeping byte payloads separate from control data.
Changes:
- Adds postcard wire types and explicit control-data lengths, and removes FlatBuffers schemas, generated code, and dependencies.
- Updates snapshot compatibility to ABI 10 and goldens v10.0, with exact-bit float assertions.
- Adds decoder fuzzing and updates communication, security, and development documentation.
| File | Description |
|---|---|
src/tests/rust_guests/Cargo.lock |
Updates guest serialization dependencies. |
src/schema/host_function_details.fbs |
Removes the host-function collection schema. |
src/schema/host_function_definition.fbs |
Removes the host-function definition schema. |
src/schema/guest_trace_data.fbs |
Removes the trace schema. |
src/schema/guest_log_data.fbs |
Removes the log schema. |
src/schema/guest_error.fbs |
Removes the guest-error schema. |
src/schema/function_types.fbs |
Removes the value and type schemas. |
src/schema/function_call.fbs |
Removes the call schema. |
src/schema/function_call_result.fbs |
Removes the result schema. |
src/schema/all.fbs |
Removes the schema entry point. |
src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs |
Selects goldens v10.0. |
src/hyperlight_host/tests/sandbox_host_tests.rs |
Checks exact float bits after round trips. |
src/hyperlight_host/src/sandbox/snapshot/tripwires.rs |
Pins ABI 10 and references wire-format tests. |
src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs |
Sets snapshot ABI 10. |
src/hyperlight_host/src/sandbox/snapshot/file_tests.rs |
Updates snapshot ABI expectations. |
src/hyperlight_host/src/sandbox/outb.rs |
Passes headers into message decoders. |
src/hyperlight_host/src/mem/virtq/codec.rs |
Reads header-sized control data and uses vector encoding. |
src/hyperlight_host/src/mem/shared_mem.rs |
Updates decoder terminology in safety comments. |
src/hyperlight_host/src/mem/mgr.rs |
Integrates postcard encoding and header-based decoding. |
src/hyperlight_host/src/error.rs |
Removes FlatBuffers-specific error variants. |
src/hyperlight_host/Cargo.toml |
Removes the FlatBuffers dependency. |
src/hyperlight_host/benches/benchmarks.rs |
Adapts codec benchmarks to vector buffers. |
src/hyperlight_guest/src/transport/context.rs |
Integrates the guest-side encoding and framing changes. |
src/hyperlight_guest/Cargo.toml |
Removes the FlatBuffers dependency. |
src/hyperlight_guest_capi/Cargo.toml |
Removes the FlatBuffers dependency. |
src/hyperlight_guest_bin/Cargo.toml |
Removes the FlatBuffers dependency. |
src/hyperlight_common/src/transport.rs |
Adds control length to the 16-byte header. |
src/hyperlight_common/src/lib.rs |
Removes the generated FlatBuffers module declaration. |
src/hyperlight_common/src/flatbuffers/mod.rs |
Removes generated module exports. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/return_value_generated.rs |
Removes generated return-value bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/return_type_generated.rs |
Removes generated return-type bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/parameter_value_generated.rs |
Removes generated parameter-value bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/parameter_type_generated.rs |
Removes generated parameter-type bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/open_span_type_generated.rs |
Removes generated span-opening bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/log_level_generated.rs |
Removes generated log-level bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/log_event_type_generated.rs |
Removes generated log-event bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/key_value_generated.rs |
Removes generated trace-field bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/host_function_details_generated.rs |
Removes generated host-function collection bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/host_function_definition_generated.rs |
Removes generated host-function definition bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/hlvoid_generated.rs |
Removes generated void bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/hlulong_generated.rs |
Removes generated unsigned 64-bit bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/hluint_generated.rs |
Removes generated unsigned 32-bit bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/hlstring_generated.rs |
Removes generated string bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/hllong_generated.rs |
Removes generated signed 64-bit bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/hlint_generated.rs |
Removes generated signed 32-bit bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/hlfloat_generated.rs |
Removes generated float bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/hlexternalbytes_generated.rs |
Removes generated external-byte bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/hldouble_generated.rs |
Removes generated double bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/hlbool_generated.rs |
Removes generated boolean bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/guest_start_type_generated.rs |
Removes generated guest-start bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/guest_log_data_generated.rs |
Removes generated guest-log bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/guest_event_union_type_generated.rs |
Removes generated event-union bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/guest_event_type_generated.rs |
Removes generated event-type bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/guest_event_generated.rs |
Removes generated guest-event bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/guest_error_generated.rs |
Removes generated guest-error bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/function_call_type_generated.rs |
Removes generated call-type bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/function_call_result_type_generated.rs |
Removes generated result-type bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/function_call_result_generated.rs |
Removes generated call-result bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/function_call_generated.rs |
Removes generated function-call bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/error_code_generated.rs |
Removes generated error-code bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/edit_span_type_generated.rs |
Removes generated span-editing bindings. |
src/hyperlight_common/src/flatbuffers/hyperlight/generated/close_span_type_generated.rs |
Removes generated span-closing bindings. |
src/hyperlight_common/src/flatbuffer_wrappers/wire.rs |
Defines postcard wire types and encoding helpers. |
src/hyperlight_common/src/flatbuffer_wrappers/mod.rs |
Registers wire encoding and ABI tests. |
src/hyperlight_common/src/flatbuffer_wrappers/host_function_details.rs |
Removes FlatBuffers serialization conversions. |
src/hyperlight_common/src/flatbuffer_wrappers/host_function_definition.rs |
Removes FlatBuffers definition conversions. |
src/hyperlight_common/src/flatbuffer_wrappers/guest_log_level.rs |
Adds serde support for log levels. |
src/hyperlight_common/src/flatbuffer_wrappers/guest_log_data.rs |
Uses postcard for logs and adds a round-trip test. |
src/hyperlight_common/src/flatbuffer_wrappers/guest_error.rs |
Removes generated error-code conversions. |
src/hyperlight_common/src/flatbuffer_wrappers/codec.rs |
Updates external-value documentation terminology. |
src/hyperlight_common/Cargo.toml |
Replaces FlatBuffers with serde and postcard. |
README.md |
Removes the schema-directory listing. |
Justfile |
Updates fuzz targets and removes FlatBuffers tooling recipes. |
fuzz/README.md |
Documents control-decoder fuzzing. |
fuzz/fuzz_targets/guest_trace_decode.rs |
Fuzzes trace decoding and event round trips. |
fuzz/fuzz_targets/control_decode.rs |
Fuzzes call, result, and log decoding. |
fuzz/Cargo.toml |
Registers the decoder fuzz targets. |
flake.nix |
Removes FlatBuffers development tooling. |
docs/virtio-host-guest-communication.md |
Documents postcard framing and external bytes. |
docs/snapshot-versioning.md |
Documents ABI 10 and wire-format enforcement. |
docs/security.md |
Describes serde/postcard decoding at the trust boundary. |
docs/security-guidance-for-developers.md |
Updates compatibility and message-validation guidance. |
docs/README.md |
Removes the FlatBuffers guide link. |
docs/how-to-use-flatbuffers.md |
Removes the obsolete generation guide. |
CHANGELOG.md |
Records the breaking serialization change. |
Cargo.lock |
Updates workspace serialization dependencies. |
.github/workflows/ValidatePullRequest.yml |
Runs decoder fuzz targets during PR validation. |
.github/workflows/Fuzzing.yml |
Updates the fuzzing target matrix. |
.github/workflows/DailyArm64.yml |
Updates the trace-fuzzer name in a comment. |
.github/copilot-instructions.md |
Removes the obsolete schema-directory listing. |
Review effort: Balanced
| ### Host-Guest Communication (Serialization and Deserialization) | ||
|
|
||
| All communication between the host and the guest is done through a shared memory buffer. Messages are serialized and deserialized using [FlatBuffers](https://flatbuffers.dev/). To minimize attack surface area, we rely on FlatBuffers to formally specify the data structures passed to/from the host and guest, and to generate serialization/deserialization code. Of course, a compromised guest can write arbitrary data to the shared memory buffer, but the host will not accept anything that does not match our strongly typed FlatBuffer [schemas](../src/schema). | ||
| All communication between the host and the guest is done through a shared memory buffer. Messages are serialized with [serde](https://serde.rs) into the [postcard](https://docs.rs/postcard) format, using strongly typed Rust definitions in `hyperlight-common`. The host copies guest data out of shared memory before decoding it. Decoding rejects malformed input and trailing bytes, and checks every length against fixed limits before allocating. A compromised guest can write arbitrary data to the shared memory buffer, but the host accepts only messages that decode into these types. |
There was a problem hiding this comment.
Fixed. Lengths are bounded by the received message, not fixed limits.
Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>
e474c07 to
91219d2
Compare
Benchmark ResultsMeasured commit: kvm / amd (Linux) (🌟 5.19x)Top improvements
Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
kvm / intel (Linux) (🌟 4.22x)Top improvements
Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
mshv3 / amd (Linux) (🌟 4.91x)Top improvements
Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
mshv3 / intel (Linux) (🌟 4.93x)Top improvements
Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
hyperv-ws2025 / amd (Windows) (🌟 2.40x)Top improvements
Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
hyperv-ws2025 / intel (Windows) (🌟 2.20x)Top improvements
Benchmark Resultsfunction_call_codec
payload_allocation
sandboxes
slot_pool
snapshot_files
virtq_readonly
virtq_readwrite
Reported by |

Implements HIP 0008 (#1999).
Details not covered by the HIP:
FunctionCall,ParameterValueandReturnValuemap to private wire types, because their byte payloads travel separately.GuestLogData,GuestEventandReturnTypederive serde and are encoded directly.abi_tests.rspins the encoded bytes of every message type. A failure explains that the change breaks snapshots and points todocs/snapshot-versioning.md.-0.0.fuzz_control_decode.fuzz_guest_trace_decodereplaces the trace size estimator target.estimate_flatbuffer_capacityand the FlatBuffersHyperlightErrorvariants.Follow-ups: borrow decoded strings, reuse encode buffers (maayybe, need to be careful about security) (#671), remove the
chunkedflag (#1987), and renameflatbuffer_wrappers.Fixes #1607